Risk Structures: Concepts, Purpose, and the Causality Problem

Risk Structures: Concepts, Purpose,and the Causality Problem

Mario GleirscherUniversity of York, UKJune 26, 2019

Shonan, JP

Part I

Risk-aware Systems:Abstraction by Example

4.0 / Gleirscher / Shonan, JP/ June 26, 2019 52

http://creativecommons.org

Example: Air-traffic Collision Avoidance System (TCAS)

Example: Safe Autonomous Vehicle (SAV)



Risk Mitigation / Intervention / Enforcement

Approach: Activesafety monitors,enforcementmonitors

System/ProcessModel

MDP,LTS, HA

RiskStructure

ActiveMonitor

Testmodel

ActiveMonitor

Implementation

MonitoredProcess

abstracts

fromabstracts

from

synthesisedfrom conform

sto

recognises/

enforces

property

RQ: How to build a mitigation monitor? / Which model to use?/ Which abstraction? / What do we need to verify?



Example: Air-traffic CollisionAvoidance System (TCAS)

Example: Traffic Collision Avoidance System (TCAS) Ego

Risk Model R

Process Model P

abstractsfrom

p1

p4e.g. red ”1`severity1`benefit ą c

p6

rl,uq

movΛ: coll

1´Λ: fin

1 Risk Factor

0coll

tp1, p6ustart

colltp4u

collisiont��mov{ecollu

t?u

Σztmov{ecolluΣztmov{finu

2 Risk Factorsbefore

0ncoll

tp1, p3,p4, p6u

startncoll

tp2, p5u

near-collisiont��tmov{encollu

t��tmov{finu

Σzttmov{encolluΣzttmov{finu

p1start

p2

p3

p4 rl4,u4q

p5 rl5,u5q

p6

mov Λ2 :encoll

1´ Λ2 : fina1a2

(TCAS move)tmov Λ5 : encoll

Λ4: ecoll

a3Λ6 : fin

a5

a40ncoll, 0coll

ncoll, 0coll 0ncoll, coll

ncoll, coll

e ncoll��ecoll

ecoll e ncoll

mncoll

“

tmov

{fin

��m “

mov{fin

ěm

Risk Space




Risk Model R

Process Model P

abstractsfrom

p1


p6

rl,uq

movΛ: coll

1´Λ: fin

1 Risk Factor

0coll

tp1, p6ustart

colltp4u


t?u



0ncoll

tp1, p3,p4, p6u

startncoll

tp2, p5u


t��tmov{finu


p1start

p2

p3

p4 rl4,u4q

p5 rl5,u5q

p6

mov Λ2 :encoll

1´ Λ2 : fina1a2


Λ4: ecoll

a3Λ6 : fin

a5

a40ncoll, 0coll


ncoll, coll

e ncoll��ecoll

ecoll e ncoll

mncoll

“

tmov

{fin

��m “

mov{fin

ěm

Risk Space




Risk Model R

Process Model P

abstractsfrom

p1


p6

rl,uq

movΛ: coll

1´Λ: fin

1 Risk Factor

0coll

tp1, p6ustart

colltp4u


t?u



0ncoll

tp1, p3,p4, p6u

startncoll

tp2, p5u


t��tmov{finu


p1start

p2

p3

p4 rl4,u4q

p5 rl5,u5q

p6

mov Λ2 :encoll

1´ Λ2 : fina1a2


Λ4: ecoll

a3Λ6 : fin

a5

a40ncoll, 0coll


ncoll, coll

e ncoll��ecoll

ecoll e ncoll

mncoll

“

tmov

{fin

��m “

mov{fin

ěm

Risk Space



Example: Safe Autonomous Vehicle(SAV)



System/ProcessModel

MDP,LTS, HA

RiskStructure

ActiveMonitor

Testmodel

ActiveMonitor

Implementation

Monitored

Process

abstracts

fromabstracts

from


sto

recognises/

enforces

property




SAV: Low Level Vehicle Dynamics

r 9p “ 0,9v “ 0s

v “ 0^ a “ 0

start

halt

r 9p “ v,9v “ as

0 ă v ă l^ a ą 0

accelr 9p “ v,9v “ 0s

v “ l^ a “ 0

drive atmax speed

r 9p “ v,9v “ as

0 ă v ă l^ a ď 0

decel

aą0

v = l

aă0

aą0

aď0

v = 0 ‖r 9α “ 0s

α “ 0start

neutral

r 9α “ f s0 ă y ă 5

turnleft

r 9α “ 0s

α ‰ 0

movestraight

r 9α “ f s´5 ă y ă 0 turn

right

y ą0

^|v|

ą0

9α“0

^α

‰0

yą0

^|v|

ą0

9α“0

^α

“0

yă0

^|v|

ą0

9α“0

^α

‰0

y ă0

^|v|

ą0

α“0

^9α

“0

Longitudinal dynamics LoDdrive

halt

accel

decel

driveat maxspeed

Lateral dynamics LaD(relative to route segment)

neutral

turnleft

turnright

movestraight

Overall low-level dynamics: drive ‖ LaD



SAV: Situational Perspective of Urban Driving

Mode model of the driving activity:

basic || supplyPower

drive

driveAtL4 || requestTakeOverByDr driveAtL1 || operateVehicle

driveAtLowSpeed

driveAtL4Generic

exitTunneldriveThroughCrossing

parkWithRemote

autoOvertake driveAtL1Generic

manuallyOvertake

autoLeaveParkingLot

manuallyPark

leaveParkingLot

steerThroughTrafficJam

halt

start

Integration withlow leveldynamics:

In each mode,verify contract:inv ^ pre ñ

wppdrive ‖ LaD,inv ^ postq



SAV: Risk Identification and Assessment

Knowledge sources for risk/hazard identification, e.g.

• accident reports• domain experts• situation/activity model• local dynamics model• control system architecture• control software

Analysis techniques, e.g.

• hazard identification: FHA, PHL, …• process/scenario analysis: HazOp, LOPA, BA, STPA, …• causal reasoning: ETA, FMEA, FTA, Bowties, …



SAV: Situational Perspective of Urban Driving

Mode model of the driving activity:

basic || supplyPower

drive

driveAtL4 || requestTakeOverByDr driveAtL1 || operateVehicle

driveAtLowSpeed

driveAtL4Generic

exitTunneldriveThroughCrossing

parkWithRemote

autoOvertake driveAtL1Generic

manuallyOvertake

autoLeaveParkingLot

manuallyPark

leaveParkingLot

steerThroughTrafficJam

halt

start

Risk factors(Yap script):

1 HazardModel for "drive"{

3 OC alias "on occupiedcourse"

;5 CR alias "increased

collision risk";

7 CC alias "on collisioncourse"

;9 ICS alias "inevitable

collision state";

11 Coll alias "actualcollision"

;13 ES alias "perception

system fault";

15 }



Risk Structures: Tool Support and Recent Publications

nColl

Coll

0

prvnCollcenColl

eColl

1 OperationalSituation "generic" {}

3 ControlLoop "Robot" for "generic" {emgBr alias "Emergency Brake";

5 }

7 HazardModel for "generic" {nColl alias "near-collision"

9 mitigatedBy (PREVENT_CRASH.emgBr)direct;

11 Coll alias "collision"requires (nColl)

13 mishap;}

From Hazard Analysis to Hazard MitigationPlanning: The Automated Driving Case

Mario Gleirscher(B) and Stefan Kugele

Technische Universitat Munchen, Munich, Germany{mario.gleirscher,stefan.kugele}@tum.de

Abstract. Vehicle safety depends on (a) the range of identified hazardsand (b) the operational situations for which mitigations of these hazardsare acceptably decreasing risk. Moreover, with an increasing degree ofautonomy, risk ownership is likely to increase for vendors towards regula-tory certification. Hence, highly automated vehicles have to be equippedwith verified controllers capable of reliably identifying and mitigatinghazards in all possible operational situations. To this end, available meth-ods for the design and verification of automated vehicle controllers haveto be supported by models for hazard analysis and mitigation.

In this paper, we describe (1) a framework for the analysis and designof planners (i.e., high-level controllers) capable of run-time hazard iden-tification and mitigation, (2) an incremental algorithm for constructingplanning models from hazard analysis, and (3) an exemplary applicationto the design of a fail-operational controller based on a given control sys-tem architecture. Our approach equips the safety engineer with conceptsand steps to (2a) elaborate scenarios of endangerment and (2b) designoperational strategies for mitigating such scenarios.

Keywords: Risk analysis · Hazard mitigation · Safe state · Controllerdesign · Autonomous vehicle · Automotive system · Modeling · Planning

1 Challenges, Background, and Contribution

Automated and autonomous vehicles (AV) are responsible for avoiding mishapsand even for mitigating hazardous situations in as many operational situationsas possible. Hence, AVs are examples of systems where the identification (2a)and mitigation (2b) of hazards have to be highly automated. This circumstancemakes these systems even more complex and difficult to design. Thus, safetyengineers require specific models and methods for risk analysis and mitigation.

As an example, we consider manned road vehicles in road traffic with anautopilot (AP) feature. Such vehicles are able to automatically conduct a rideonly given some valid target and minimizing human intervention. The followingAV-level (S)afety (G)oal specifies the problem we want to focus on in this paper:

SG: The AV can always reach a safest possible state σ wrt. the hazardsidentified and present in a specific operational situation os .

c© Springer International Publishing AG 2017C. Barrett et al. (Eds.): NFM 2017, LNCS 10227, pp. 310–326, 2017.DOI: 10.1007/978-3-319-57288-8 23

L. Bulwahn, M. Kamali, S. Linker (Eds.): First Workshop onFormal Verification of Autonomous Vehicles (FVAV 2017).EPTCS 257, 2017, pp. 75–90, doi:10.4204/EPTCS.257.8

c© M. GleirscherThis work is licensed under theCreative Commons Attribution License.

Run-Time Risk Mitigation in Automated Vehicles:A Model for Studying Preparatory Steps

Mario Gleirscher

Faculty of InformaticsTechnical University of Munich

Munich, [email protected]

We assume that autonomous or highly automated driving (AD) will be accompanied by tough assur-ance obligations exceeding the requirements of even recent revisions of ISO 26262 or SOTIF. Hence,automotive control and safety engineers have to (i) comprehensively analyze the driving process andits control loop, (ii) identify relevant hazards stemming from this loop, (iii) establish feasible auto-mated measures for the effective mitigation of these hazards or the alleviation of their consequences.

By studying an example, this article investigates some achievements in the modeling for thesteps (i), (ii), and (iii), amenable to formal verification of desired properties derived from potentialassurance obligations such as the global existence of an effective mitigation strategy. In addition, theproposed approach is meant for step-wise refinement towards the automated synthesis of AD safetycontrollers implementing such properties.

1 Introduction

For many people, driving a car is a difficult task even after guided training and many years of drivingpractice: This statement gets more tangible when driving in dense urban traffic, complex road settings,road construction zones, unknown areas, with hard-to-predict traffic co-participants (i.e., cars, trucks,cyclists, pedestrians), bothersome congestion, or driving a defective car. Consequently, hazards such asdrivers misjudging situations and making poor decisions have had a long tradition. Hence, road vehicleshave been equipped with many sorts of safety mechanisms, most recently, with functions for “safetydecision support” [7], driving assistance, and monitor-actuator designs, all aiming at reducing operationalrisk or making a driver’s role less critical. In AD, more highly automated mechanisms will have tocontribute to risk mitigation at run-time and, thus, constitute even stronger assurance obligations [7].

In Sec. 1.1, we introduce basic terms for risk analysis and (run-time) mitigation (RAM) for automatedvehicles (AV) as discussed in this work. Sec. 1.2 elaborates some of these assurance obligations.

1.1 Background and Terminology

According to control theory, a control loop L comprises a (physical) process P to be controlled anda controller C, i.e., a system in charge of controlling this process according to some laws defined byan application and an operator [20]. The engineering of safety-critical control loops typically involvesreducing hazards by making controllers safe in their intended function (SOTIF), resilient (i.e., toleratedisturbances), dependable (i.e., tolerate faults), and secure (i.e., tolerate misuse).

In automated driving, the process under consideration is the driving process which we decomposeinto a set of driving situations S , see Sec. 2. Taxonomies of such situations have been published in, e.g.

154

8.4 Strukturen für die Gefahren-erkennung und -behandlung in autonomen Maschinen

Dr. Mario GleirscherDepartment of Computer Science, University of York und Fakultät für Informatik, Technische Universität München

In diesem Abschnitt werden hochautomatisierte – insbesondere autonom handelnde – Maschinen (AM) wie zum Beispiel autono-me mobile Roboter betrachtet. Man erwartet von solchen Syste-men, dass deren Regelungen in Gefahrensituationen ebenso nützliche Handlungsalternativen bieten wie im Normalbetrieb. Ausgehend von dieser Problemstellung wird nun eine werkzeug-gestützte Herangehensweise

i. für die Modellierung von Gefahrensituationen sowie

ii. für die Bewertung der Plausibilität und Vollständigkeit solcher Modelle

anhand eines Beispiels aus dem Bereich des automatisierten Fahrens (AF) diskutiert.

1 Motivation

Im Rahmen der gefahrenreduzierenden Absicherung von Syste-men ist es eine Herausforderung, möglichst starke Sicherheits-eigenschaften für autonome, hochautomatisierte Maschinen schon zum Entwurfszeitpunkt festzulegen und Regler solcher Maschinen für die Einhaltung dieser Eigenschaften zur Laufzeit zu entwerfen. Im Folgenden werden formale Strukturen, welche für die Modellbildung und später für die detaillierte Entwicklung von Reglern hilfreich sind, besprochen. Die Motivation, solche Strukturen zu nutzen, resultiert aus

§ Bestrebungen, die Risikobewertung und den Verlässlich-keitsnachweis für allgemeine Systemklassen durch speziali-sierte Modellbildung zu unterstützen (siehe Kapitel 4/Bertsche et al.),

1 | Vgl. Kugele et al. 2017.2 | Vgl. Alexander et al. 2009.3 | Vgl. McDermid 2001 und Kumamoto 2007 diskutieren „As Low As Reasonably Practicable“ (ALARP).4 | Vgl. Schnieder/Schnieder 2009, Schnieder/Schnieder 2008, Schnieder/Drewes 2008.5 | Vgl. Lund et al. 2011.

§ Erfahrungen in der Architekturabsicherung komplexer einge-betteter Systeme1 und

§ der Beobachtung, dass einige Empfehlungen für die Gewähr-leistung von AM-Sicherheit nicht eindeutig und vollständig sind.2

2 Hintergrund

In diesem Abschnitt werden einige Begriffe aus der Literatur und Vorarbeiten des Autors dargestellt, auf denen die spätere Diskus-sion aufbaut.

2.1 Allgemeine Grundlagen

In der Risikoanalyse wird bewertet, inwiefern eine Gefahrenquel-le (Aggressor) ein Risiko für ein Schutzziel (zum Beispiel Safety, Security, körperliche Unversehrtheit) eines Schutzobjekts (im Englischen: asset) darstellt und inwieweit ein Schutzmechanis-mus (auch Beschützer, Sicherheitsfunktion) dieses Risiko wenigs-tens auf ein akzeptables Restrisiko3 reduzieren kann. Häufig wird dazu eine Menge wahrscheinlicher Szenarien in Form von poten-ziell unendlichen Ursache-Wirkungs-Ereignisketten betrachtet, wobei die ultimativen und unerwünschten Auswirkungen als Un-glücks-, Unfalls- oder Schadensereignis und alle Ereignisse auf diesem Wege als Zusammensetzung von potenziell verursachen-den Faktoren beschrieben werden. Hierzu wird weiter unten von Kausalfaktoren und -strukturen gesprochen. Ein kompatibler Be-griffsrahmen wird in den Kapiteln 3/Schnieder und Schnieder, 5/Beyerer und Geisler sowie 7.1/Vieweg ausführlicher behandelt.

Dieser vielfach diskutierte Begriffsrahmen lässt sich auf ganz un-terschiedliche Bereiche anwenden, zum Beispiel technische An-lagen, IT-Systeme, in der Patientenbehandlung im Krankenhaus, im unternehmerischen Projektmanagement, in Arbeitsprozessen im Hochbau oder an Flughäfen (siehe Kapitel 8.1/Wolf und Lichte sowie 8.2/Deutschmann et al.). Je nach Art des Schutz-ziels und -objekts gibt es spezifische Herangehensweisen zur RA sowie verschiedene Bezeichnungen, wie zum Beispiel funktiona-le Sicherheit in der Mechatronik und Automatisierungstechnik4 oder Cyber Security für stark vernetzte IT-Systeme.5 Regelmäßig wird versucht, Erkenntnisse aus verschiedenen Arbeitsfeldern

arX

iv:s

ubm

it/26

6338

0 [

cs.S

E]

23

Apr

201

9

RISK STRUCTURES: TOWARDS ENGINEERING RISK-AWARE

AUTONOMOUS SYSTEMS

A PREPRINT

Mario GleirscherComputer Science Department, University of York, York, UK∗

April 23, 2019

ABSTRACT

Inspired by widely-used techniques of causal modelling in risk, failure, and accident analysis, thiswork discusses a com positional framework for risk modelling. Risk models capture fragments ofthe space of risky events likely to occur when operating a machine in a given environment. More-over, one can build such models into machines such as autonomous robots, to equip them with theability of risk-aware perception, monitoring, decision making, and control. With the notion of arisk factor as the modelling primitive, the framework provides several means to construct and shaperisk models. Relational and algebraic properties are investigated and proofs support the validity andconsistency of these properties over the corresponding models. Several examples throughout thediscussion illustrate the applicability of the concepts. Overall, this work focuses on the qualitativetreatment of risk with the outlook of transferring these results to probabilistic refinements of thediscussed framework.

Keywords Causal modelling · risk · analysis · modelling · safety monitoring · risk mitigation · robots · autonomoussystems

Contents

1 Introduction 2

1.1 Abstractions for Machine Safety and Risk Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Contributions and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Background 7

2.1 Notions of Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 The Risk of Undesired Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Formal Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Risk Elements 9

∗Correspondence and offprint requests to: Mario Gleirscher, Univerity of York, Deramore Lane, Heslington, York YO10 5GH,UK. e-mail: [email protected] work is supported by the Deutsche Forschungsgemeinschaft (DFG) under Grants no. GL 915/1-1 and GL 915/1-2. c© 2019. This manuscript is made available under the CC-BY-NC-ND 4.0 licensehttp://creativecommons.org/licenses/by-nc-nd/4.0/.Reference Format: Gleirscher, M.. Risk Structures: Towards Engineering Risk-aware Autonomous Systems (April 23, 2019).Unpublished working paper. Department of Computer Science, University of York, United Kingdom.



Risk Factors (Basic Phase Model)

0f

f

fef …endangermentofn

nominaloperation…

mf …mitigationrecovery… mfr

mfd …direct mitigation

endanger-ment … ef

ofe…endangeredoperation

ofm …mitigated operation

Purposes:• Modelling primitive for risk space exploration• Semantics of basic events in DFTs or DFRTs• Synthesis of local enforcement monitors



SAV: Situational Risk Space

CCOCICSCR

CollCR

CCOCCR

CR

OCICSColl

OCCR

CCICSCR CCOCCollCCCollCR

CCOCICSColl

CCICSColl

ICS

ICSColl OCColl

OCICSCR

CCOC

0

OCICSCollCR

CCICS

ICSCollCR CCOCICS

ICSCR

CC

OCCollCR

OCICSCCCR

CCOCCollCR

Coll

CCColl

CCOCICSCollCR

CCICSCollCR

OC

eVehicle

ICS eVehicle

OC

eVehicle

OC

eVehicle

CC

eVehicle

CC

eVehicle

Coll

eVehicle

CR

eVehicle

ICS

eVehicle

CC

eVehicle

ICS

eVehicle

CR

eVehicle

OC

eVehicle

Coll

eVehicle

CR

eVehicle

Coll

eVehicle

CR

eVehicle

CC

eVehicle

ICS

eVehicle

Coll

eVehicle

Coll

eVehicle

CR

eVehicle

ICS

eVehicle

ICS

eVehicle

OC

eVehicle

CReVehicle

OC

eVehicle

Coll eVehicle

ICS

eVehicle

Coll eVehicle

ICSeVehicle

CR

eVehicle

OC eVehicle

CR

eVehicle

OC

eVehicle

OC

eVehicle

ICSeVehicle

CC

eVehicle

CC eVehicle

ICS

eVehicle

OC

eVehicle

ICS

eVehicle

CC

eVehicle

ColleVehicle

ICS

eVehicle

OC

eVehicle

Coll

eVehicle

ICS

eVehicle

CC

eVehicle

Coll eVehicle

CR

eVehicle

OC

eVehicle

CR

eVehicle

Coll

eVehicle

CC

eVehicle

CC

eVehicle

OC

eVehicle

ICS eVehicle

CReVehicle

Coll

eVehicle

Coll

eVehicle

OC

eVehicle

OC

eVehicle

CR eVehicle

CC

eVehicle

Coll

eVehicle

CR

eVehicle

CC

eVehicle

OC

eVehicle

CR

eVehicle

CC

eVehicle

Coll eVehicle

CR

eVehicle

CC

eVehicle

CR

eVehicle

ICS

eVehicle

ICS

eVehicle

OC

eVehicle

CC

eVehicle

CC

eVehicle

Coll



SAV: Situational Risk Space (zoomed)

CCOCICSCR

CollCR

CCOCCR

CR

OCICSColl

OCCR

CCICSCR CCOCCollCCCollCR

CCOCICSColl

CCICSColl

ICS

ICSColl OCColl

OCICSCR

CCOC

0

OCICSCollCR

CCICS

ICSCollCR CCOCICS

ICSCR

CC

OCCollCR

OCICSCCCR

CCOCCollCR

Coll

CCColl

CCOCICSCollCR

CCICSCollCR

OC

eVehicle

ICS eVehicle

OC

eVehicle

OC

eVehicle

CC

eVehicle

CC

eVehicle

Coll

eVehicle

CR

eVehicle

ICS

eVehicle

CC

eVehicle

ICS

eVehicle

CR

eVehicle

OC

eVehicle

Coll

eVehicle

CR

eVehicle

Coll

eVehicle

CR

eVehicle

CC

eVehicle

ICS

eVehicle

Coll

eVehicle

Coll

eVehicle

CR

eVehicle

ICS

eVehicle

ICS

eVehicle

OC

eVehicle

CReVehicle

OC

eVehicle

Coll eVehicle

ICS

eVehicle

Coll eVehicle

ICSeVehicle

CR

eVehicle

OC eVehicle

CR

eVehicle

OC

eVehicle

OC

eVehicle

ICSeVehicle

CC

eVehicle

CC eVehicle

ICS

eVehicle

OC

eVehicle

ICS

eVehicle

CC

eVehicle

ColleVehicle

ICS

eVehicle

OC

eVehicle

Coll

eVehicle

ICS

eVehicle

CC

eVehicle

Coll eVehicle

CR

eVehicle

OC

eVehicle

CR

eVehicle

Coll

eVehicle

CC

eVehicle

CC

eVehicle

OC

eVehicle

ICS eVehicle

CReVehicle

Coll

eVehicle

Coll

eVehicle

OC

eVehicle

OC

eVehicle

CR eVehicle

CC

eVehicle

Coll

eVehicle

CR

eVehicle

CC

eVehicle

OC

eVehicle

CR

eVehicle

CC

eVehicle

Coll eVehicle

CR

eVehicle

CC

eVehicle

CR

eVehicle

ICS

eVehicle

ICS

eVehicle

OC

eVehicle

CC

eVehicle

CC

eVehicle

Coll



Causality (Lewis 1973)

Definition (Counterfactual Conditional)A 2Ñ C is nonvacuously true iff C holds at all the closestA-worlds.

What are the closest A-worlds?



SAV: Risk Identification and Assessment Yap script

OperationalSituation "drive"2 {

include "envPerc";4 }

6 ControlLoop "Vehicle" for "drive"{

8 replan alias "Slow-down || re-planroute";

brake alias "Standard brake";10 swerve alias "Short-term circumvention

of obstacle";EB alias "Emergency brake";

12 accel alias "Accelerate";airbag alias "Front airbag";

14 }

16 HazardModel for "drive"{

18 OC alias "on occupied course"mitigatedBy (PREVENT_CRASH.replan)

20 direct;

22 CR alias "increased collision risk"requires (OC)

24 deniesMit (OC)

excludes (OC)26 mitigatedBy (PREVENT_CRASH.EB)

;28 CC alias "on collision course"

requires (CR)30 deniesMit (CR,OC)

excludes (CR,OC)32 mitigatedBy (PREVENT_CRASH.swerve)

;34 ICS alias "inevitable collision state"

requires (CC)36 excludes (CC,CR,OC)

causes (Coll)38 mitigatedBy (PREVENT_CRASH.EB)

;40 Coll alias "actual collision"

requires (ICS)42 excludes (CC,CR,OC,ICS,ES)

mitigatedBy (ALLEVIATE.airbag)44 mishap

;46 ES alias "perception system fault"

excludes (CC,CR,OC,ICS)48 deniesMit (OC,CC)

;50 }




0

ES

CC

OC

CR

Coll

eVehicle

ES

eVehicle

CollICS

eVehicle

CR

eVehicle

CC

eVehicle

OC

eVehicle

ES

eVehicle

ES

eVehicle

ES

Approach: LOPA/BA tocreate chain of possibleinterventions




0

ES

CC

CC

ES

CR

Coll

OC

Coll

CR

eES

eES

eES

mCCe

eCR

prvCCc

mColle

eES

eCC

eOC prvOCc

prvCRc

attCollalv

eES

rES

eCollICS

mCRe

fopESfb

eES


Layered interventionpattern for SAV obstacleavoidance

Nice side-effect: Usepattern as enhancedphase model for similarrisk factors



Taxonomy of Mitigations

OverallSafety

SafetyConstraints

FaultContainment

Fail-Safe

FaultDetection

FaultAvoidance

PreventiveSafety

PassiveSafety

Error-CorrectingCodes

RedundancyFail-Operational

Limp-Home

Fail-Silent

Fail-Secure

ConditionMonitoring

Checking

Pre-CrashAssistance

ObstacleAvoidance

Limiter

Protection

partiallyrealizes

Override

fullyrealizes

Non-Interference

SanityCheck

VigilanceCheck

Warning

Masking

VotingDegradation

Recovery

Rollback

Repair

SimplicitySubstitution

Replication Diversity

Barrier

Comparison

Shutdown

Interlocking

Separation

Stabilization

Reconfiguration

Notification

Filter



Taxonomy of MitigationsOverallSafety

SafetyConstraints

FaultContainment

Fail-Safe

FaultDetection

FaultAvoidance

PreventiveSafety

PassiveSafety

Error-CorrectingCodes

RedundancyFail-Operational

Limp-Home

Fail-Silent

Fail-Secure

ConditionMonitoring

Checking

Pre-CrashAssistance

ObstacleAvoidance

Limiter

Protection

partiallyrealizes

Override

fullyrealizes

Non-Interference

SanityCheck

VigilanceCheck

Warning

Masking

VotingDegradation

Recovery

Rollback

Repair

SimplicitySubstitution

Replication Diversity

Barrier

Comparison

Shutdown

Interlocking

Separation

Stabilization

Reconfiguration

Notification

Filter



SAV: Situational Risk Space (Monitor Candidate)

0

ES

CC

CC

ES

CR

Coll

OC

Coll

CR

eES

eES

eES

mCCe

eCR

prvCCc

mColle

eES

eCC

eOC prvOCc

prvCRc

attCollalv

eES

rES

eCollICS

mCRe

fopESfb

eES


= Specification of a validsafe system

expected order violatedÑ lack of observability,incomplete monitor,context mismatch?



Risk Structures: Templates for Causal Reasoning

A risk structure R is valid iff for

s, s1, s2, s3 P RpFq, s3 P Mishaps, e,m, e1 P Σ˚

@s, s3 P RpFq, t P R Ds1, s2 P RpFq,m P Σ˚ : t “ eme1^

s s’ s” s”’e m e’

Definition (Mitigation from Counterfactual Perspective)m is mitigation of cause c ‰ s2 of a mishap s3 iff e1 gets unlikely.(s2, e1 form the counterfactual.)

Proof obligations for each t: Check that, from s,

1. s2 is actual cause of s3,

2. s1 is recognisable,

3. from s1, m reduces s2.4.0 / Gleirscher / Shonan, JP/ June 26, 2019 75




System/ProcessModel

MDP,LTS, HA

RiskStructure

ActiveMonitor

Testmodel

ActiveMonitor

Implementation

MonitoredProcess

abstracts

fromabstracts

from


sto

recognises/

enforces

property




Documents

Risk Structures: Concepts, Purpose, and the Causality Problem