UAS Integration Risk Federal Aviation Assessment Air

Presented to: Committee on Assessing the Risks of UAS Integration

Presented by: Federal Aviation Administration

Date: Wednesday, September 27, 2017

Federal AviationAdministration

UAS Integration Risk AssessmentAir Traffic Organization

FEDERAL AVIATIONADMINISTRATION

What is a Safety Management System?

29/27/2017 SMS Overview

Risk Management

Safety Policies and Standards

Communication, Values, and

CultureData, Analysis, and Reporting

SMS

An integrated collection of processes, policies, procedures, and programs used to assess, define, and manage safety risk in the provision of air traffic control

and navigational services


What Does SMS Provide?


Continuous monitoring,

assessment, & mitigation

Common framework to

identify & address safety hazards

and risks

Communication/ management of

potential & actual risks in the

system

Data-Informed, Expertise-Driven Management


What are the SMS Components?


Safety Assurance

Safety Promotion

Safety Risk

Management

Safety Policy


What is the SRM Process?

DIAAT: The formalized approach used by a panel of experts and affected stakeholders to identify issues, evaluate

their criticality, and determine a means to minimize or eliminate their impact

59/27/2017 SRM Process Overview

Treat the Risk

Identify the Hazards

Analyze the Risk

Assess the Risk

Describe the System


Treat the Risk


Analyze the Risk

Assess the Risk

Describe the System

Describe the System

9/27/2017 SRM Process Overview: Describe the System

6


What is the 5M Model?

A tool used to describe the

system, operation, or procedures being added or

changed

79/27/2017 SRM Process Overview: Describe the System

(hu)Man/ Person

MachineManagement

Media or Environment

Mission


Treat Risk


Analyze Risk

Assess Risk

Describe the System

Identify Hazards

9/27/2017 SRM Process Overview: Identify Hazards 8


Any real or potential condition that can cause injury, illness, or death to people; damage to or loss of a system, equipment, or property; or damage to the environment. A prerequisite to an accident or incident.

What is a Hazard?

99/27/2017 SRM Process Overview: Identify Hazards

Preliminary Hazard List/

What If Analysis

Procedural Changes (system or

operational)

Functional Hazard Analysis

Acquisition or modification of

equipment

Bow-Tie Diagram

All applications when it is difficult to determine hazard or

cause/effect relationship

Tool/Technique Best Use


Treat Risk


Analyze Risk

Assess Risk

Describe the System

Analyze Risk

9/27/2017 SRM Process Overview: Analyze Risk 10


What Is Risk?The composite of predicted severity and likelihood of the potential effect of a hazard, before any of the

proposed mitigations are implemented

119/27/2017 SRM Process Overview: Analyze Risk

Severity

• The consequence or impact of a hazard’s effect (outcome) in terms of degree of loss or harm (less severe to more severe)

• While assessed first, determination of severity is independent of likelihood

Likelihood

• The estimated (predicted) probability or frequency, in quantitative or qualitative terms, of a hazard’s effect (outcome)

• Definitions are tailored to ATC Operations, Flight Procedures, and Systems Engineering


Assess Risk

Treat Risk


Analyze Risk

Assess Risk

Describe the System

9/27/2017 SRM Process Overview: Assess Risk 12


What are Risk Levels?

139/27/2017 SRM Process Overview: Assess Risk

Risk levels are assessed for each hazard based on severity and likelihood

• Must be mitigated to a Medium or Low risk prior to implementation

High: Unacceptable

Risk

• May be implemented but safety requirements are recommended to increase the safety margin

Medium: Acceptable

Risk

• May be implemented (but safety requirements are recommended)

• Must have at least one safety performance target

Low: Acceptable

Risk

Seve

rity

/ Lik

elih

ood


Treat Risk

Treat Risk


Analyze Risk

Assess Risk

Describe the System

9/27/2017 SRM Process Overview: Treat Risk 14


• Document all hazards and document and verify low-risk hazards at least once• Document a plan to implement safety requirements for all risk levels identified in the final

Hazard Analysis Worksheet (HAW)• Record monitoring activities

How do we Treat Risk?

159/27/2017 SRM Process Overview: Treat Risk

• Identify and evaluate means to manage the risk or reduce it to an acceptable level• The four risk management strategies are: Risk Control, Risk Avoidance, Risk

Transfer, and Risk Assumption

Identify feasible risk management

options

• The risk that is estimated to exist after the safety requirements are implemented, or after all avenues of risk mitigation have been explored.

Determine predicted

residual risk

• Identify measurable goals used to verify the predicted residual risk of a hazard• Determine appropriate metrics• Consider controls and safety requirements• Pre-SRM panel data analysis serves as the basis for comparison against the

post-implementation metrics

Define safety performance

targets

Develop a monitoring plan

1

2

3

4


Evaluating UAS Risk Using SRM

169/27/2017 Evaluating UAS Risk Using SRM


Scoping UAS Integration

9/27/2017 17Evaluating UAS Risk Using SRM

• UAS risk analysis efforts focused on the following:– Line-of-sight operations– Operations beyond visual line of sight– Commercial uses and operations (i.e., the Pathfinder

Program)– Certificate of Authorization (COA) processes– Impact on Airspace Class (e.g., operations near and

around airports and large population centers)


Past UAS Safety Studies

– COA process for UAS low-altitude operations under Section 333 (200 feet Above Ground Level (AGL))

– COA process for UAS low-altitude operations under Section 333 at or below 400 feet AGL

– Class B

– Class A and High E

– Class A and High E– Class C– Class D– Class E– Class G– Conditions and limitations for

allowing operations in Class E Surface Area

9/27/2017 18

• SRM panels convened for each class of airspace and Pathfinders for the safe integration and testing of UAS in the National Airspace System (NAS)

Evaluating UAS Risk Using SRM


Common Hazards Across Safety Analyses Lost Link - The loss of real-time command and control (C2) data link. If the link between the Pilot in Command (PIC) and the UAS is lost, the Unmanned Aircraft (UA) utilizes pre-programmed contingency (i.e., lost link) procedures until the link is re-established or until the UAS ends the flight in a safe manner

Fly Away - Interruption or loss of the control link, or when the pilot is unable to affect control of the aircraft and, as a result, the UA is not operating in a predicable or planned manner

Difficulty to be Seen by Manned Aircraft - Due to size of UA (e.g., Quad Copters or small fixed-wing), speed, body composition, and markings



Loss of Visual Line of Sight - The PIC has lost visual contact with the UA as required by 14 Code of Federal Regulations Section 91.113Loss of Communications Between Pilot and Observer because of radio failure and they are not co-located

Common Hazards Across Safety Analyses (cont.)



Worst Credible Effect

1973 MOR in 2016144 NMAC in 20162 MAC in 25 years

Most UAS operations -NMAC is the Worst Credible Effect


1.00E-11

1.00E-10

1.00E-09

1.00E-08

1.00E-07

1.00E-06

1.00E-05

1.00E-04

1.00E-03

1.00E-02

1.00E-01

1.00E+00MOR NMAC MAC

Minimal 5 Minor 4 Major 3 Hazardous 2 Catastrophic 1

FrequentA

ProbableB

RemoteC

ExtremelyRemote

D

ExtremelyImprobable

E

SMS Risk Matrix


Current UAS Effects and Severity Classifications

22

UAS Hazard Severity ClassificationNote: Severities related to ground-based effects apply to movement areas only.

Minimal5

Minor4

Major3

Hazardous2

Catastrophic1

CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING:

Und

er 5

5 po

unds

and

100

mph

or s

low

er Discomfort to those on the ground

Low Risk Analysis Event severity, two or fewer indicators fail

Non-serious injury to three or fewer people on the ground

A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins

UAS crash obstructing Runway

Medium Risk Analysis Event severity, three indicators fail

Incapacitation to Unmanned Aircraft System crew

Non-serious injury to more than three people on the ground

Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 100 feet

High Risk Analysis Event severity, four indicators fail

Proximity of less than 100 feet to a manned aircraftSerious injury to persons other than the Unmanned Aircraft System crew

A collision with a manned aircraft

Fatality or fatal injury to persons other than the Unmanned Aircraft System crew



Current UAS Effects and Severity Classifications

UAS Hazard Severity ClassificationNote: Severities related to ground-based effects apply to movement areas only.

Minimal5

Minor4

Major3

Hazardous2

Catastrophic1

CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING:

Ove

r 55

poun

ds o

r fas

ter t

han

100

mph

Discomfort to those on the ground

Loss of separation leading to a Measure of Compliance greater than or equal to 66 percent

Low Risk Analysis Event severity, two or fewer indicators fail

Non-serious injury to three or fewer people on the ground

Medium Risk Analysis Event severity, three indicators fail

Non-serious injury to more than three people on the ground

A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins

Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 500 feetUAS crash obstructing Runway

High Risk Analysis Event severity, four indicators fail

Incapacitation to Unmanned Aircraft System crew

Proximity of less than 500 feet to a manned aircraftSerious injury to persons other than the Unmanned Aircraft System crew

A collision with a manned aircraft

Fatality or fatal injury to persons other than the Unmanned Aircraft System crew



Pros and Cons of NMACs as Credible Effects

NMACs are reported by controllers and pilots

NMACs are tracked and categorized using existing data collection

It is difficult or impossible to determine the cause of the NMAC involving a UAS

UAS operators cannot be identified

The existence and criticality of a NMAC is subjective based on its current definition (requires third-party verification)



Monitoring UAS Encounters

25

• Quarterly, AJI collects data on UAS encounters; demographics include:– Altitude– Nearest ATC facility– Proximity to manned aircraft– Type of manned aircraft involved– UAS position in relation to manned aircraft– Class of airspace– Phase of flight (manned aircraft)– Severity of NMAC



Trend in UAS Encounters (April–July 2017, FY17 Q3)

0

2

4

6

8

10

12

14

16

18

20

Total

Linear (Total)



FY17 Trends in UAS Encounters


Data set – all 500ft NMACs (1/1/17 - 9/19/17)• 41% are 100ft NMACs• 5% occurred under 500ft altitude• 53% occurred over 3000ft altitude

Reports Identifying AirspaceClass A: 0.5% Class D: 14%Class B: 34% Class E: 19%Class C: 15% Class G: 5%


Future of UAS Risk Assessment

289/27/2017 Future of UAS Risk Assessment


UAS Integration Risk

The primary goal of the ATO is to prevent collision with manned aircraft in controlled airspace, while FAA

lines of business/other agencies are responsible for risks not related to air traffic


Collision with manned aircraft,

people on the ground, or other

UAS

Damage to property

Environment Issues


Target Level of Safety

309/27/2017

• The NAS is a very safe system, but it is not risk-free– The probability of collision is greater than zero– The goal is to maintain a low probability of collision, but

how low?• Most air navigation service providers have a target

level of safety.– The ATO’s minimum acceptable level of safety for a

catastrophic outcome is 1E-9• Recently, the ATO adopted an Acceptable Level of

Safety concept for rare operations, such as commercial space efforts

Future of UAS Risk Assessment


Quantitative Risk Assessment• Risk should be assessed in relation to its

“proximity” to collision– All incidents and accidents should be analyzed with

respect to their probability of collision• Need for a Risk Continuum concept

– Employ modeling: • Use objective data • Analytical approaches; geometry and density• Minimize reliance on subject matter expert input

by employing modeling (e.g., Monte Carlo modeling)



Factors in Modeling• Not all collisions are created equal. This is

dependent upon:– UAS size– UAS speed– Vulnerability of aircraft

• “Credit”: Controls and mitigations should be accounted for (e.g., sense and avoid)

• “Penalty”: All credible sources of errors should be accounted for (e.g., human error)



Current Modeling Effort• Under the Low-Altitude Authorization and Notification

Capability (LAANC) initiative, the FAA has developed maps with pre-approved flight zones and maximum altitudes for operating drones near airports

• With the near-term focus of supporting LAANC, Volpe is working on a quantitative model to examine small UAS operating in controlled airspace around airports– Input data: Manned aircraft tracks– Output 1: Collision probability – Output 2: Risk-Adjusted Altitude

• Collision probability is negligible below, and non-negligible above, Risk-Adjusted Altitude

– Output 3: Sample Size• Used to assign confidence level to Outputs 1 and 2



Manned aircraft

Flight “swath” modeled by aircraft frontal profile area extruded over distance traveled

Airport

Small UAS randomlylocated in airspace

Relevant airspace examined over a given period of time, number of aircraft, etc.

Collision Probability

Aircraft frontal profiles


Ratio of manned aircraft “swath” volumes to total airspace volumeLength of analysis time period

Unmitigated Collision Probability per Unit Time =


Manned aircraft

• Controlled airspace divided into 1-min x 1-min grid volumes• Risk-Adjusted Altitude is defined as ”200 feet below routine

operations” in each grid volume:

Maximum altitudeof grid volume (400 feet)

Altitude of routine manneda/c operations in grid volume

Ground levelRisk-AdjustedAltitude

Highest Risk

200 feetSafety Buffer

Operatewith

caution

Risk-Adjusted Altitude



Routine Operations• Volpe’s model is in the early stages of development• Risk-Adjusted Altitude is calculated on a grid-by-grid

basis, rounded down to the nearest 50 feet in each grid, and presented two ways

• “Worst-case” = Lowest altitude reached manned aircraft tracks (points within tracks) over analysis time period, minus 200-foot safety buffer

• “Fifth Percentile below ceiling”: 1,200 feet selected as ceiling, below which is considered shared airspace– Filter out lowest 5 percent as “outliers” or “rare operations”– 200 feet below fifth percentile of points below 1,200 feet in each

grid volume



Challenges for Risk Analysis


Insufficient data collection to

establish reliable performance monitoring

Lack of procedures for

UAS

Communication issues (e.g., non-

standard terminologies)

Difficult to quantify

mitigations

Lack of licensing requirements to

maintain a level of design reliability

Insufficient operator

proficiency


Takeaways

38

ATO’s primary safety focus is on preventing collision between manned aircraft


Modeling approaches focused on collision probability to determine the true risk from UAS operations in the NAS.

Improved data (collected and simulated) for monitoring performance targets.

Future StateSafety and operational experts have used the SRM process to assess UAS risk potential.

AJI is collecting data on UAS encounters with manned aircraft.

Current State


Back-Up Slides

9/27/2017 39Back-Up Slides


SRM Panels• Carry out the SRM process and produce safety

documents with risks and recommendations to mitigate them.

• Stakeholders and Process Participants:– Safety engineers– ATC experts– Pilots– Bargaining unit representatives– Data analysts– Industry subject matter experts

409/27/2017 Back-Up Slides


SRM Process


SRM Document

If No Hazards Are Identified

Review Current System

Review Proposed Change or Existing

Issue

Develop Preliminary Hazard List

Develop Hazard

Analysis Worksheet

Develop Monitoring

Plan


Link Between Cause, Hazard and Effect

Cause(s)

Ice Hazard

Possible Effect


9/27/2017 Back-Up Slides 42


Preliminary Hazard List

• Hazard identification tool that lists potential hazards

• The result of brainstorming (may include hazards, causes, effects, and sometimes system state or controls)

• Used to list all possible hazards within scope• Stepping stone to HAW


Ultimately, the list will be a combination of hazards, causes, and effects that will be later categorized in the SRM process. Within/out of scope is not addressed at this time.



HAW

• What is a HAW?– An analysis tool used to document the risk assessment of the

system or change– Inputs are developed from a PHL (or other Hazard

Identification tool)

• Why complete a HAW?– Required as part of the SRM process– Links identified hazards to controls and risk mitigation

activities– To document risks, hazards, system states, and safety

requirements




HAW: Template (Key Definitions)

(1)Hazard ID

(2)Hazard

Description

(3)Causes

(4)System State

(5)Controls

(6)Control

Justification

(7)Effects

(Credible Effects)

(8)Severity

(9)Severity

Rationale

(10)Likelihood

(11)Likelihood Rationale

(12)Initial Risk

(13)Safety

Requirements

(14)Organization

Responsible for Implementing

Safety Requirements

(15)Predicted Residual

Risk

(16)Safety

Performance Targets

* Refer to Quick Reference Guide for Additional Information.




(1)Hazard ID

(2)Hazard Description

(3)Causes

(4)System State

ABC-01 Ice accumulation on the sidewalk (slipping hazard)

Freezing rain

Snow

November-March

Temperatures below 32 °F

HAW Ice Example: Identify Hazards



Controls: Guidelines

• Understand the impact of the control• Must be associated with the change, hazard, cause

and system state• Cite the specific version, paragraph and/or section

number(s) when using FAA Orders• Include information explaining how the control

mitigates the risk


Analyze Risk


Effects (Credible Effects)

• Credible effect refers to the reasonable expectation that the assumed combination of conditions that define the system state will occur within the operational lifetime of a typical ATC system.

• Hazard assessments consider all credible effects (outcomes).

• Less severe effects may pose a higher risk than the worst credible effect.


Analyze Risk


SMS Likelihood Table

49

Operations: Expected Occurrence Rate (per operation / flight hour / operational hour)

Quantitative (ATC / Flight Procedures / Systems Engineering)

Frequent (A) (Probability) ≥ 1 per 1,000

Probable (B) 1 per 1000 > (Probability) ≥ 1 per 100,000

Remote (C) 1 per 100,000 > (Probability) ≥ 1 per 10,000,000

Extremely Remote (D) 1 per 10,000,000 > (Probability) ≥ 1 per 1,000,000,000

Extremely Improbable (E) 1 per 1,000,000,000 > (Probability) ≥ 1 per 1014


Analyze Risk


(5)Control(s)

(6)Control(s) Justification

(7)Effects

(8)Severity

(9)Severity Rationale

(10)Likelihood

(11)Likelihood Rationale

HAZARD: ABC-01

Current building safety/security plan

ABC regulations

Intervention by facility personnel

BLD PLN 001

ABC Reg 002

Facilitate personnel intervene by placing warning signs and spreading salt/ice melting agents.

Sprained Wrist

Head injury

Major (3)

Hazardous (2)

Slipping on ice can cause joint injuries, bodily harm, muscle

strains

Probable (B)

Extremely Improbable

(E)

10,000 people use the sidewalks, and approximately 5 injuries occur due to slips on ice, giving a rate of 5x10-4 per operation (one per 2000).

Over the past 10 years 36 head injuries have been reported. Estimate that a typical person makes about 100 trips per winter on foot. Total population of US is about 300 million, so rate is 36 / (10*100*3x108) = 1.2x10-11 per operation.”

Slipping on ice can cause accidental

head injury.

HAW Ice Example: Analyze Risk


Analyze Risk


HAW Ice Example: Assess Risk


(7)Effects

(8)Severity

(10)Likelihood

(12)Initial risk

Initial Hazard Risk

Sprained WristMajor

(3)Probable

(B)

Head injuryHazardous

(2)

Extremely Improbable

(E)

3B -High

2E - Medium

3B - High

Assess Risk


HAZARD: ABC-01

ABC-01

HAW Ice Example: Initial Risk


Assess Risk


HAW Ice Example: Treat Risk


(13)Safety

Requirements

(14)Organization Responsible

forImplementing Safety

Requirements

(15)Predicte

dResidual

Risk

(16)Safety

Performance Targets

HAZARD: ABC-01

Establish sand buckets at each end of the sidewalk. Custodian will sand the sidewalk each hour when the hazard is present.

Post facility-wide warning of icing

Each employee will receive a briefing on the alternate route to follow if the hazard is present.Facility management will budget the cost of sidewalk anti-icing materials. Jane Smith, Director

Jeene Smith, ABC Custodian

Joe Smith, ABC Manager3C

Medium

Fewer than two falls each winter due to slipping

on ice.

Treat Risk


ABC-01

HAW Ice Example: Predicted Residual Risk


Treat Risk


55

Monitoring Plan Template


Documents

UAS Integration Risk Federal Aviation Assessment Air