Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN

Functional Safety beyond

ISO26262 for Neural Networks in

Highly Automated Driving

Autonomous Driving Meetup #5

MAN Track Forum, Munich

27th of March 2018

André Roßbach, Tim Heinemann, Florian Bogenberger

Motivation

Copyright © exida.com 2000-201827/03/2018 2

ISO26262

CNN, AI, ML

Agenda

• What is beyond ISO26262 incl. 2nd Edition? (today)

• How can NNs become "safe"?

• „Probably correct“ – Is this sufficient?

• How to measure „Safety“? – Metrics for NNs

• What is beyond NNs? (today)

27/03/2018 Copyright © exida.com 2000-2018 3

Structure of the Implementation


Classic Development Neural Networks

Paradigm ShiftStructure homomorph

to System

Structure disparat

to System

Using „ISO-Methods“ for NNs?

Basic Aspect for

ISO26262

Static SW-

Algorithms

Parameter

Learning

Learning of Rules

(NNs, etc.)

Structur of

Implementation

Similarity to logical-

functional structure ☺

Fault-Injection Test ☺..

Statische Analysis ☺ ☺..

Prove of „Non-Behavior“ ☺ ☺..

Requirements Coverage

Requirements Tracing ☺..

Structural Coverage ☺

Training

Some methods according to ISO26262 work

no longer for neural networks


Tool Chain Aspects


Tool chain complexity & degree of

automation will exceed by far today´s

development tool chains

ISO26262-8 11 “Confidence in the

use of Software Tools”

ISO26262-7 “Production & Operation”

... requires enhancements

Enhancement: Tool Safety Concept


Agenda







Approach and Systematics


ISO26262 Mapping Matrix


ISO26262

Structure

Extensions and

adaptations

map

map

Machine Learning Safety Concept


Machine learning

safety concept

Specification of safety

requirements

for machine learning

Dataset safety requirements

• ML Algorithm

• Performance Measures

• Avoidance of unintended Behavior

• Detection of Implausibility

• Error Detection & Mitigation

• …

... a simplifying abstraction of real error effects ...

... intended to enable systematic analysis

in reality fault propagation is quite complex, but ...

... frequently different faults lead to similar errors

sometimes seem more pessimistic than reality ... but ...

reality is much more „creative“ than the human brain can

foresee


Fault Model – What it is ...

Fault Models are Key for Safety Analysis

(Static & Dynamic)


Fault Model – Applied for NNs

Von Chrislb - Erstellt von Chrislb, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=224561

• Weight too high

• Weight too low

• Weight corrupt

• …

„Simple“ Approach: Fault Model on Structural Level

https://commons.wikimedia.org/w/index.php?curid=224561

Usage of different solutions to prevent, reduce or

detect failures

homogenous - assumes using an identical safety

element multiple times ( the „same“ twice)

heterogenous - assumes using different safety

elements to fulfill the same safety requirements

Why we need Diversity ...


Diversity is one of the key methods to detect faults

beyond the limits of “single brain” human

reasoning.

ML: Diversity for “free” ...


Voting on results

Input

Difference

Machine Learning is well suited to realize diversity ...

... different ...

• architectures

• training data

• ML approaches

... come “for free”.

Agenda







Failure without Fault?

ISO26262 simple model: Fault, error, failure chain


… could there be a failure without a fault?

… are human faults just "stupidity"?

???

Key Insight: We need to understand how ”Hypothesis” work ...

essential for human intelligent behavior

... ISO26262 has zero idea about this

Extend your thinking beyond “faults” ...

... recognize the power of hypotheses ... improve hypothesis

... likewise

for NNs ...

Agenda







Structural Coverage 1

ISO26262: Metric to identify shortcomings in test

cases, inadequacies in requirements or

unintended functionality

Meaning

– Find unneeded code

– Find needed code … but missing test

– Find needed code … but missing requirement


Structural Coverage 2

Structural Coverage for SW (ISO26262):– Statement Coverage

– Branch Coverage

– Modified Condition/Decision Coverage (MC/DC)

… for Neural Networks:– Measure Degree of Neuron Activations

– Possible Formula:

𝑆𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑎𝑙 𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒 % =𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑒𝑑 𝑁𝑒𝑢𝑟𝑜𝑛𝑠

𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑁𝑒𝑢𝑟𝑜𝑛𝑠∗ 100

𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑒𝑑 𝑁𝑒𝑢𝑟𝑜𝑛 = max 𝑎𝑖) − min(𝑎𝑖 > 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑


Structural Coverage 3 - Example

Adapted for Neural Networks

– Observation of activation of nodes during test run

– Example


…

Variance

(example only)

Dangerous Confusion Metric (DCM) 1


Real Class

Pedestrian Car Bus Cyclist Normal Street

PredictedClass

Pedestrian 568 0 0 0 0

Car 50 1056 0 25 0

Bus 150 10 746 15 0

Cyclist 30 0 0 198 0

Normal Street 200 300 700 800 3254

ASIL QM

ASIL A

ASIL B

ASIL C

ASIL D

Apply Safety Weight Factors

according to the assigned

ASIL of a misclassification

Traditional Confusion Matrix (example)

Safety Confusion Matrix

Weighted mis-

classification based on

impact (violation of a

safety goal)

Real Class


Predicted Class


Car 5 1056 32 5 5

Bus 15 5 746 3 7

Cyclist 6 7 3 198 12

Normal Street 2 3 7 8 3254

Real Class


Predicted Class


Car 5 1056 32 5 5

Bus 15 5 746 3 7

Cyclist 6 7 3 198 12

Normal Street 2 3 7 8 3254

0

2

5

10

100

Dangerous Confusion Metric (DCM) 2


Real Class


Predicted Class


Car 50 1056 0 25 0

Bus 150 10 746 15 0

Cyclist 30 0 0 198 0

Normal Street 200 300 700 800 3254

𝐷𝐶𝑀𝐴𝑆𝐼𝐿 𝑆𝐺 Dangerous Confusion Metric for a given ASIL or safety goal respectively.

𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺Number of false neg. classifications per class and ASIL or safety goal respectively.

𝑛𝑇𝑃𝐴𝑆𝐼𝐿 𝑆𝐺 Number of true pos. classifications per class and ASIL or safety goal respectively.

𝐷𝐶𝑀𝐴𝑆𝐼𝐿 𝑆𝐺 = 1 −σ𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺

σ𝑛𝑇𝑃𝐴𝑆𝐼𝐿 𝑆𝐺 +σ𝑛𝐹𝑁𝐴𝑆𝐼𝐿 𝑆𝐺

True Pos. and False Neg.

Classifications per SG

(example for ASIL B SG)

Agenda







Unintended Behaviour

Unintended or unknown behaviour could lead to

violation of a safety goal/requirement

For neural networks unintended behavior

is often very close

(see adversarial attacks)

Identify the potential unintended behaviours

Testing won´t solve the problem ...

... unintended behaviour must be “excluded by design”


Error Detection and Mitigation


unexpected pedestrian – safety

anomaly unless visible before

Wrong decision

Correct decision

Key Insight: 100% correct decision not always possible

=> Calculate multiple variants in parallel

Essential: Plausibility Checks


implausible location of

traffic light – anomaly

must be detected by NN

Conclusion: We need PROGRESS ..

... for ISO26262 ... for NNs & ML

Concepts

ML Safety Concept Systematics to handle

Functional SafetyTool Safety Concept

Link to Safety Requirements

Algorithms

& Design

Probability-based Algorithms Avoid unintended Behaviour

by Design

Hypothesis-based Prediction & Decision

Verification

& Validation

Detect Implausibility

Error Detection & Mitigation

Stress & Boundary Testing

AnalysisSafety Analysis Methods (Static & Dynamic)

Fault Models for Structure & Behavior

MetricsSafety Metrics suitable for

NNs, ML & probability-based Algorithms


What do think? ....


Many Thanks for your [email protected]

[email protected]

[email protected]

excellence in dependable automation

27/03/2018 35

Documents

Functional Safety beyond ISO26262 for Neural Networks in ... · Functional Safety beyond ISO26262 for Neural Networks in Highly Automated Driving Autonomous Driving Meetup #5 MAN