Comparison of accident risk assessment by event sequence … · 2008-11-06 · Comparison of accident risk assessment by event sequence analysis versus Monte Carlo simulation Sybert

Comparison of accident risk assessment by event sequence analysis versus Monte Carlo simulationComparison of accident risk assessment by event sequence analysis versus Monte Carlo simulation

Sybert Stroeve, Henk Blom, Hans de Jong (DFS), Jelmer Scholte

EUROCONTROL Safety R&D Seminar, Southampton, UK, 22-24 October 2008

EUROCONTROL Safety R&D Seminar, Southampton, UK, 22-24 October 2008 2

Sequential accident modelsSequential accident models

Accident = Sequence of ordered events, such as failures or malfunctions of humans or machines

Examples: event trees, domino theory

Predominantly used in reliability engineering and risk assessment

S

F

S

FS

FEvent tree

“Pivotal”Event

S

F

S

FS

F

S

F

S

F

S

FS

F

S

F

Causes Consequences

HAZARD Effect A

Effect B

Effect C

Effect DFault tree


Systemic accident modelsSystemic accident models

Accident = Emergent from the performance variability of a joint cognitive system, as a result of complex interactions and unexpected combinations of actions

Examples: STAMP, FRAM, TOPAZ

Recent development and yet sparsely used in safety assessments

Human1

Human3System2

System3

criticalinteractions

Human2

System1

System4


Limitations of sequential accident modelsLimitations of sequential accident models

Sequential accident models may not be adequate for the complexity of modern socio-technical systems (Hollnagel 2004, yesterday; Leveson 2004; Sträter 2005):

Difficult to account for complex multi-agent interactionsNo dynamic, non-linear behaviour Only performance at the level of event probability

What does this mean in a practical risk assessment?


Safety assessment of active runway crossing operationSafety assessment of active runway crossing operation

Crossing operation runway 18C Schiphol

Part of development cycles (Scholte et al. 2008)

Broad scopecontrollerspilotsrunway incursion alert systemactive stopbarR/T systemsground radar crossing procedure

Good visibility conditions

09

18C

36C

36C

C1 8

0 9

A22A23

A24

A25

A26

A27

A28

A21

N5

N9

A19

A19

A1

W7

W9

W8

W10

W10

W6

W5

W3

W4

W2

W1

B

B

B

A

A

A

S

S

W

W

V

VM

V

VP4

P5

A


Safety assessment stepsSafety assessment steps

Determine operation1

Assess risk tolerability6

Assessseverity4

Identify safety

bottlenecks7

Assess frequency5

Construct scenarios3Identify

hazards2

Identify objective0

Decision making

Operational

1. sequential models 2. systemic models

development


Conflict scenarioConflict scenario

Hazardous situation:

Pilot of taxiing aircraft thinks to be on a normal taxiway / Pilot of taxiing aircraft starts crossing without contacting runway controller (e.g. by misunderstanding ground controller)

Conflict:

Aircraft is erroneously taxiing across the runway while another aircraft is taking off


Frequency assessment by event treeFrequency assessment by event tree

Pilots recognitionController recognitionRIASS alertCommunication

earlymediumlate

Aircraft crossing while it should not

Pilots recognition

early

Controller recognition

early

RIASSearly

Comm.early

Pilots recognition

medium


medium

RIASSmedium

Comm.medium

Pilots recognition

late


late

Comm.late Result

Early resolution

Early resolution

Early resolution

Medium resolution

Medium resolution

Medium resolution

Medium resolution

Medium resolution

Late resolution

Late resolution

Late resolution

Late resolution

Late resolution

Late resolution

Accident

Accident

Accident

Accident

Accident

Accident

No aircraft in take-off

No conflict


Event probabilitiesEvent probabilities

0.60.4Communication leads to effective resolution at late stage0.750.5Controller recognizes conflict at late stage0.990.9Pilots recognize and resolve conflict at late stage0.80.6Communication leads to effective resolution at medium stage0.990.9Alert effectively warns controller at medium stage0.40.2Controller recognizes conflict at medium stage0.990.9Pilots recognize and resolve conflict at medium stage0.90.8Communication leads to effective resolution at early stage0.990.95Alert effectively warns controller at early stage0.20.1Controller recognizes conflict at early stage0.70.5Pilots recognize and resolve conflict at early stage0.750.75No aircraft in take-off

Upper bound

Lowerbound

Event ProbabilityEvent


Event tree resultsEvent tree results

7.3 E-56.5 E-8Accident

1.1 E-31.6 E-5Late resolution

2.8 E-28.0 E-3Medium resolution

2.4 E-12.2 E-1Early resolution

7.5 E-17.5 E-1No conflict

Upper boundLower bound

Conditional probabilityEvent tree result


Systemic accident model:Multi-agent stochastic dynamic modelSystemic accident model:Multi-agent stochastic dynamic model

Key aspects of agents, e.g.SA / task performance of operator Flight phase / aircraft performance

Modes within key aspects, e.g.Task: monitoring / alert reaction Flight phase: taxi / take-off

Dynamics within modes, e.g.Task performance timeTake-off acceleration profile

InteractionsBetween modesBetween key aspects of an agentBetween agents


Risk assessment includes MC simulation + bias & uncertainty assessment Risk assessment includes MC simulation + bias & uncertainty assessment


Conditional accident risk resultsConditional accident risk results

2.2 E-6(6.5 E-8 – 7.3 E-5)Event tree

1.7 E-4 (4.1 E-6 – 7.3 E-4)Systemic accident model

Conditional accident riskMethod

What are the causes of the differences?


Additional MC simulation resultsto support analysis of differencesAdditional MC simulation resultsto support analysis of differences

8.9 E-2nonono3

2.3 E-4noyesyes2

1.7 E-4yesyesyes1

ATCoPF taking-offaircraft

PF taxiingaircraft

Conditionalaccident risk

Agent in the monitoring loopCase


Comparison by simple event treeComparison by simple event tree

yes

yes

yes

no

no

no



yes

yes

yes

no

no

no

A

DB C

E



yes

yes

yes

no

no

no

A

DB C

EP(B|A)

8.9 E-12.5 E-1(2.5 E-1 – 2.5 E-1)

Systemic approachSequential approach


Event: No aircraft in take-offEvent: No aircraft in take-off

Relevant aspects are well known and can be evaluated accurately

Interwoven dynamic aspects are hard to judge accurately

Relevant aspects in MC simulationsTiming of take-off vs. taxiingTaxi speedSpeed profile of take-off runLift-off pointInitial climb angleRunway geometry

Directly assessed by safety expert

P(B|A)

8.9 E-12.5 E-1(2.5 E-1 – 2.5 E-1)




yes

yes

yes

no

no

no

A

DB C

EP(C|A,B)

2.6 E-33.9 E-4(3.0 E-5 – 5.0 E-3)



Event: Pilots resolve conflictEvent: Pilots resolve conflict

Significant uncertainty in risk level

Bias & uncertainty assessment pinpoints uncertain aspects

Significant uncertainty in risk level

Relevant aspects in MC simulationsMonitoring performance of pilotsConflict recognition by pilotsConflict reaction by pilotsDeceleration profiles of aircraftMC simulation aspects of event B

Separate pilots flying

Combination of event probabilities for pilots’ resolution at early, medium or late stage

Performance of pilots of both aircraft is combined

P(C|A,B)

2.6 E-33.9 E-4(3.0 E-5 – 5.0 E-3)




yes

yes

yes

no

no

no

A

DB C

EP(D|A,B,C)

7.4 E-12.3 E-2(8.6 E-3 – 5.9 E-2)



Event: Controller resolves conflictEvent: Controller resolves conflict

ATC performance aspects contribute only to a small extent to uncertainty

MC simulations support effective analysis of dependencies

Lack of event tree analysis support has led to neglect of dependencies and overestimation of controller contribution to conflict resolution

Relevant aspects in MC simulationsMonitoring by controllerConflict recognition by controllerAlert reaction by controllerCommunication by controllerMC simulation aspects of B, C

Combination of event probabilities for controller recognition, alerts and communication at early, medium or late stage

P(D|A,B,C)

7.4 E-12.3 E-2(8.6 E-3 – 5.9 E-2)



Conflict scenario timeline example 1Conflict scenario timeline example 1

start TO RTO hold

brake

Aircraft A

start taxi brake holdAircraft B

see conflict start RTOPilot A

see conflictPilot B

see conflict RT

ATCoActive

Alert


Conflict scenario timeline example 2Conflict scenario timeline example 2

start TO RTO

brake

Aircraft A

start taxiAircraft B

see conflict start RTOPilot A

see conflict brakePilot B

see conflict RT

ATCoActive

Alert



yes

yes

yes

no

no

no

A

DB C

EP(B|A)·P(C|A,B)·P(D|A,B,C)

1.7 E-4(4.1 E-6 – 7.3 E-4)

2.2 E-6(6.5 E-8 – 7.3 E-5)



Event: AccidentEvent: Accident

Effective risk analysis support by MC simulation of the dynamic and concurrent performance of interacting agents

Lack of event tree analysis support has led to neglect of dependencies and thereby to likely underestimation of the risk

MC simulations including all agents and their dynamic and concurrent performance

Bias & uncertainty assessment

Conditional accident risk by combination of probabilities of failure of events B, C and D

P(B|A)·P(C|A,B)·P(D|A,B,C)

1.7 E-4(4.1 E-6 – 7.3 E-4)

2.2 E-6(6.5 E-8 – 7.3 E-5)



DiscussionDiscussion

Argument: You took the wrong numbers in the event tree!

Reply: You may tune the event probabilities to obtain the same results in the sequential and systemic approaches.

But, how to know? No analysis support.

And then, consider having changed the probabilities such that the conditional risk reduction by the controller is low.

But what in different contextual conditions, such as reduced visibility?


MC simulation results in reduced visibilityMC simulation results in reduced visibility

10-6

10-5

10-4

10-3

10-2

Con

ditio

nal c

ollis

ion

risk

(per

take

-off)

Without RIASWith RIAS

Good visibility Reduced visibility

Dependence with pilots’ visual monitoring

ATC and RIAS make a difference:

Safety benefit


ConclusionsConclusions

TOPAZ multi-agent stochastic dynamic modelling of air traffic scenarios is a systemic approach

Risk emerges from Monte Carlo simulations addressing performance variability of interacting agentsRisk estimates account straightforwardly for contextual conditions

Event trees may represent event dependencies, but they lack analysis support to evaluate their (conditional) probabilities, and ... such analysis may be a difficult job

Results of a systemic model can be represented in an event treeSupports transparency of risk resultsRequires additional, dedicated MC simulations

Safety benefit of ATC/RIAS is low for a runway incursion in goodvisibility, but significant in reduced visibility


Questions / DiscussionQuestions / Discussion

Documents

Comparison of accident risk assessment by event sequence … · 2008-11-06 · Comparison of accident risk assessment by event sequence analysis versus Monte Carlo simulation Sybert