CAATS II C APPROACH TO AIR TRAFFIC S II - TRIMIS · This document provides guidance for safety case development in ATM R&D projects, with a focus on application to advanced operational

Project no.: FP6-2005-TREN-4-Aero- 036826

CAATS II

COOPERATIVE APPROACH TO AIR TRAFFIC SERVICES II

Instrument: CA - Coordination Action

Thematic Priority: AERO-2005-1.3.1.4h

D14: Guidance document for a typical safety case

Due date of deliverable: 06/05/2009 Actual submissi on date: 08/10/2009

Start date of project: 06/11/2006 Duration: 36 mont hs

Organisation name of lead for this deliverable: NLR

Revision: Draft

Project co-funded by the European Commission within the Sixth Framework Programme (2002-2006)

Dissemination Level

PU Public X

PP Restricted to other programme participants (including the Commission Services)

RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services)

“Cooperative Approach to

Air Traffic Services II”

Date:

Document ID:

Revision:

08/10/2009

CII-WP1.2-NLR-006-V1.9-DE-PU

Draft

D14: Guidance document for a typical safety case - 1 -

This project has been carried out under a contract awarded by the European Commission

© 2009 – All rights reserved

Document Change Log

Revision Edition Date Author Modified Sections / Pages Comments

1.9 8 October

2009

Jelmer Scholte (NLR) Henk Blom (NLR) Alberto Pasquini

(Deep Blue) Bas van Doorn (NLR)

All First public version



Date:

Document ID:

Revision:

08/10/2009


Draft




Executive summary E-OCVM, the standard European framework for validation of operational concepts in ATM R&D, has recently formalized the use of ‘cases’ in ATM R&D. These ‘cases’ have the objective to group information into common aspects in order to describe the potential of concepts under evaluation, and thereby support the key stakeholders as they make the investment and implementation decisions, along the R&D part of the lifecycle of a concept. One of the E-OCVM cases addresses safety; other example cases are for human factors, for business and for environment. Whereas the development of a safety case by an ANSP for a change to its ATM system (including humans, procedures, and technical equipment) has become common practice, safety case development in R&D has been subject of a lot of recent research. An analysis of this research is documented in this guidance document’s companion document [CAATS II D13], Parts 1 and 2. From that analysis it appears that: • experiences with developing a safety case in E-OCVM are just building up; • several needs are emerging for safety case development for advanced developments such as aimed

for by SESAR, as traditional approaches fall short; and • several new, complementary approaches are emerging that aim to address the SESAR-identified

emerging needs. This document provides guidance on safety case development in ATM R&D while these new complementary approaches are still in development, and while it is not yet fully clear how they can be integrated in tackling the multiple emerging needs of safety case development of large, SESAR-like changes. The guidance presented consists of guidance for defining safety case activities in line with the E-OCVM framework, guidance for using safety analysis, and guidance for addressing the SESAR-identified emerging needs. The guidance document provides the following key recommendations: • Experiences should be gained with emerging methods; and • Integration of the emerging methods should be continued in order to combine their strong points.



Date:

Document ID:

Revision:

08/10/2009


Draft




Table of Contents

Executive summary.................................. ....................................................................................3

1. Introduction....................................... .....................................................................................6

1.1. Background...........................................................................................................................6 1.2. Purpose.................................................................................................................................7 1.3. Relations with other ATM safety documents..........................................................................7 1.4. Document structure...............................................................................................................8

2. The European Operational Concept Validation Methodo logy ...........................................9

2.1. E-OCVM objectives and principles ........................................................................................9 2.2. Three aspects of validation..................................................................................................11 2.3. Fitting the parts together .....................................................................................................12

3. Guidance for defining safety case activities ....... ..............................................................14

3.1. Safety case development evolves with maturity ..................................................................14 3.2. Guidance regarding definitions............................................................................................14 3.3. Guidance for developing expertise ......................................................................................15 3.4. Guidance for using E-OCVM...............................................................................................15 3.5. Guidance for re-use of safety results...................................................................................17

4. Guidance for using safety analysis ................. ..................................................................18

4.1. Overview of a safety analysis process.................................................................................18 4.2. Detailed description of the stages of the safety analysis process ........................................19 4.3. Safety analysis stages needed per phase of E-OCVM ........................................................25 4.4. Guidance for the selection and further development of safety techniques ...........................26

5. Guidance for addressing SESAR-identified emerging n eeds..........................................29

5.1. Introduction .........................................................................................................................29 5.2. The need for a ‘macro’ safety case (A) ................................................................................29 5.3. The need to address safety regulations (B) .........................................................................31 5.4. The need to address the multi-stakeholder nature of advancing air traffic operations

(C).......................................................................................................................................32 5.5. The need to address the success side of a change (D) .......................................................33 5.6. The need to cover performance of human operators (E) .....................................................33 5.7. The need to identify unknown ‘emergent’ risks (F) ..............................................................35 5.8. The need to address E-OCVM requirements (G).................................................................36 5.9. The need to assess concept maturity (H) ............................................................................37 5.10. The need for managing relations between cases (I) ............................................................38

6. Concluding remarks................................. ...........................................................................41

7. References ......................................... ..................................................................................42

- 6 - D14: Guidance document for a typical safety case



1. Introduction 1.1. Background The European Operational Concept Validation Methodology (E-OCVM) is the standard European framework for validation of operational concepts in ATM Research and Development (R&D). Part of E-OCVM is the case-based approach, used along the lifecycle of a concept for grouping information about the ATM performance and behaviours of an operational concept. One of the aims of the CAATS II project, a 6th Framework Programme project funded by the European Commission, is to develop good practice documents and guidance documents for four of these E-OCVM cases: the safety case, the human factors case, the business case, and the environment case. Based on these guidance documents, the CAATS II project provides input to version 3 of E-OCVM; the version of E-OCVM available to the CAATS II project is version 2 [E-OCVM]. For more background information, one is referred to E-OCVM and [CAATS II website]. It is important to realize that the development of each case makes part of a general operational concept validation process. In order for a new operational concept to be validated, it must usually satisfy demanding objectives set in multiple performance areas, which are of importance to a wide range of stakeholders. ICAO defined 11 Key Performance Areas (KPAs) in which the performance of an operational concept is expressed [ICAO doc 9854], which have also been embraced by SESAR [SESAR D2] (cf. Figure 1-1). These KPAs are Capacity, Safety, Environment, Cost-effectiveness, Efficiency, Flexibility, Predictability, Security, Access/ Equity, Participation, Interoperability. Validation with respect to the KPA safety can also be called safety validation.

Figure 1-1: The performance of ATM is expressed in 11 Key Performance Areas (KPAs). SESAR aims

to improve ATM performance on all these KPAs (From [SESAR D2]).




1.2. Purpose E-OCVM, the standard European framework for validation of operational concepts in ATM R&D, has recently formalized the use of ‘cases’ in ATM R&D. These ‘cases’ have the objective to group information into common aspects in order to describe the potential of concepts under evaluation, and thereby support the key stakeholders as they make the investment and implementation decisions, along the R&D part of the lifecycle of a concept. One of the E-OCVM cases addresses safety; other example cases are for human factors, for business and for environment. This document provides guidance for safety case development in ATM R&D projects, with a focus on application to advanced operational and technical developments such as aimed for by SESAR. Whereas the development of a safety case by an ANSP for a change to its ATM system (including humans, procedures, and technical equipment) has become common practice, safety case development in R&D has been subject of a lot of recent research. An analysis of this research is documented in this guidance document’s companion document [CAATS II D13], Parts 1 and 2. From that analysis it appears that: • experiences with developing a safety case in E-OCVM are just building up; • several needs are emerging for safety case development for advanced developments such as aimed


emerging needs. Purpose of this guidance document is to describe how safety case development is best done in ATM R&D while these new complementary approaches are still in development, and while it is not yet fully clear whether and how they can tackle the multiple emerging needs posed by safety case development of large, SESAR-like changes. The document assumes use of E-OCVM as a starting point. Attention is given to the relations of the safety case with the operational concept development process and with other cases. E-OCVM poses specific, new requirements to safety case development in R&D, which all boil down to optimal information provision to stakeholders to enable an effective and efficient development and validation process. Typically, such information needs to be provided when concepts under validation cannot yet be stated to be safe, and when individual options need further development or redevelopment before being optimized for the future operation on all performance areas. Eventually, if a concept appears to be successful, such a safety case from R&D can become the foundation for a safety case in support of certification and implementation. Then, also the safety case definition as known from literature is fulfilled, being ‘the documented assurance of the achievement and maintenance of safety’. Primary target audience of the document is formed by managers and practitioners involved in the development and validation of new ATM concepts. This document also aims to provide input to version 3 of E-OCVM. 1.3. Relations with other ATM safety documents Several other useful documents are available in ATM safety. The relations with this guidance document of some relevant documents are as follows: • This documents companion document, [CAATS II D13] (Parts 1 and 2), provides an overview of

current and emerging practices in safety assessment in ATM R&D, and of needs that emerge safety when being applied to advanced developments such as aimed for by SESAR. It is an update of the ‘Good practices in safety assessment’ document from the first CAATS project [CAATS, D1.4 P2], but with a focus on the R&D phases.




• [CAATS Safety] provides the main results regarding safety of the first CAATS project. This includes good practices in safety assessment methodologies, safety management, safety regulation, and safety R&D.

• In FAA/ Eurocontrol Action plan 15: ATM Safety Techniques and Toolbox [AP15 Toolbox], a generic overview is given of a safety analysis/ assessment process. The current guidance document explains that throughout safety case development in R&D use is made of such safety analysis, and explains how this is done in the framework of E-OCVM.

• The current guidance document provides guidance how approaches currently already in use and newly emerging approaches can be used in safety case development in ATM R&D, with a focus on fundamental changes as aimed for by SESAR. The main approaches currently already in use are:

o The Air Navigation Services Safety Assessment Methodology [SAM]; o Eurocae’s “Guidelines for approval of the provision and use of Air Traffic Services

supported by data communications” [ED-78A]; and o The TOPAZ accident risk assessment methodology [TOPAZ].

Among the approaches that have more recently emerged are: o Safety Assessment Made Easier (SAME, [SAME PT1]), which is incorporating [SAM]; o Safety Fundamentals [Safety Fundamentals]; and o The SAFMAC (Safety validation of major changes) safety validation framework [SAFMAC].

These three emerging approaches are introduced in detail in [CAATS II D13], Part 2, Appendices V, VI, and VIII. It is noted that SAM is being incorporated in the development of SAME.

• The Safety Case Development Manual [SCDM], as part of SAM [SAM], describes how a safety case can be developed that forms the documented assurance of the achievement and maintenance of safety. It was developed with focus on use by ANSPs for changes in their ATM system (including humans, procedures, and technical equipment). Currently, its alignment with E-OCVM is being elaborated in the development of SAME. For more detailed information on this, one is referred to [CAATS II D13], Part 2, Appendix VII.

SESAR documents particularly relevant for safety case development in ATM R&D are: • [SESAR WP1.6.2/ D3] describes the early screening of concepts on Safety Fundamentals [Safety

Fundamentals] undertaken in SESAR. • [SESAR SMP] explains the foreseen safety case development approach for SESAR, including the

foreseen use of SAME [SAME PT1]. For a more recent view on this from Eurocontrol, one is referred to [CAATS II D13], Part 2, Appendix V.

• [SESAR D6] provides an overview of the SESAR Work Programme for 2008 to 2013, including the summarized foreseen safety case development approach.

1.4. Document structure The structure of this document is as follows: • Section 2 explains the principles of E-OCVM; • Section 3 provides guidance for defining safety case development activities in line with E-OCVM; • Section 4 gives guidance for the use of safety analysis; • Section 5 gives guidance for addressing the SESAR-identified emerging needs in safety case

development for advanced concepts; • Section 6 provides concluding remarks; and • Section 7 provides references.




2. The European Operational Concept Validation Meth odology This section introduces E-OCVM, based on its description in [E-OCVM]. Section 2.1 describes the main E-OCVM objectives and principles. Section 2.2 describes the three aspects of validation introduced by E-OCVM. Next, Section 2.3 describes how these are fit together. 2.1. E-OCVM objectives and principles The R&D community is continuously investigating new operational concepts that can advance ATM. These advancements include aspects such as increased throughput or capacity; reduction of environmental impact; increase in safety; and reduction of the air traffic service costs. The R&D community shall not only develop innovative ATM concepts, but also improve these concepts, and move these towards an operational status with a process of refinement and consolidation. The R&D community also has to provide adequate evidence that these concepts are able to deliver the planned enhancements while preserving or improving the overall system safety. This evidence shall demonstrate that new procedures can work according to its performance objectives in a real life environment while addressing the problems for which they were developed. An operational concept has its own lifecycle during which the concept is continuously refined and improved. Different alternative versions of the concept can be considered, especially during the initial phases of the lifecycle. One of the aims of validation is to test these different versions and enable selection of those that are most promising for the later phases of the lifecycle. Also, validation should support the process whereby the many stakeholders eventually should come to decisions to either continue development, eventually to implementation, or to stop or substantially modify developments due to some inadequacy of the overall performance or behaviour. Operational concept validation is defined as ‘the iterative process by which the fitness-for-purpose of a new system or operational concept being developed is established’. It can also be defined as the process of answering the question ‘Are we building the right system?’, whereas verification can be taken as answering the question ‘Are we building the system right?’. E-OCVM [E-OCVM] has become the reference framework for validation of ATM concepts in European R&D projects; also SESAR’s operational concept validation methodology [SESAR CVM] is based on E-OCVM. E-OCVM aims operational concept validation to be objective and transparent in order to support the decision making process where the different stakeholders have different information needs and sometimes conflicting requirements. Where applicable, stating that the system has a negative impact, or that there still remains a high level of risk, is as valid as stating where a system has a positive impact. E-OCVM is focused on the consistent provision of information on performance capability and operability. Performance of ATM concepts can be expressed in a number of key performance areas. Any solution targeted at a specific performance area will also need to be evaluated for its impact on other performance areas. One of the key performance areas is safety. Operational concept validation naturally encompasses the need for information about safety issues, because safety is the prime concern of ATM in all cases, and because safety has a direct or indirect influence on all the remaining objectives. For example, a negative impact on safety would adversely impact all other strategic objectives such as capacity and economy. Accordingly, safety analysis is helpful to identify preliminary feedback to reduce the risks associated with a new concept and provide evidence for the safety of the concept. Quantity and characteristic of useful feedback, as well as the type of evidence, can differ, depending on the concept being at the very early stages of its lifecycle or being already well consolidated and mature. Ideally, the safety analysis should thus be tailored to the concept lifecycle, and the activities should be scoped on the basis of the maturity of the concept. In Figure 2-1, E-OCVM’s Concept Lifecycle Model is introduced:




Figure 2-1: E-OCVM’s Concept Lifecycle Model

As an illustration, parallels can be drawn with the construction of a new car. Early mock-ups of ’concept cars’ could be created to produce a more aerodynamic car, these mock-ups could be considered to understand if the new concept car can really bring benefits in terms of improved "air penetration" – this could be phase V1 above. This would support decisions about moving to prototyping the various component sub-systems. Then different projects would be created to develop brakes, engines and bodywork. The overall feasibility would be evaluated and this could be the equivalent of phase V2. At some point these would again be brought together as a prototype with greater fidelity than the mock-up but still not suitable to be industrialised. This would be used to ensure compatibility of the different aspects before further engineering commenced. Such a test could be considered as a point equivalent to the end of one of the lifecycle phases shown above. The complexity of developing and moving into operation new ATM operational concepts generally exposes too many issues for one project to handle successfully. E-OCVM is based upon such development practise, and delivers the opportunity to have multiple R&D projects to design, build, test and evaluate a concept into a working application that can be industrialised. A real example of two large R&D projects contributing to the development of a concept at different levels of its maturity process is shown in Figure 2-2:




V3 - Integration

MFF experiments on ASAS spacing

G2G experiments on ASAS spacing

ASAS Spacing Concept Lifecycle

V1 - Scope V2 - Feasibility

2002 - 2006

V1 - Scope V2 - Feasibility V3 - Integration

Figure 2-2: Lifecycle of the ASAS Spacing concept with contribution of two R&D projects

2.2. Three aspects of validation E-OCVM includes three aspects of validation that, when viewed together, help provide structure to an iterative and incremental approach to concept development and concept validation: • The Concept Lifecycle Model facilitates the setting of appropriate validation objectives, the choice of

evaluation techniques, shows how concept validation interfaces with product development and indicates where requirements should be determined;

• The Structured Planning Framework facilitates programme planning and transparency of the whole process;

• The Case-Based Approach integrates many evaluation exercise results into key ‘cases’ that address stakeholder issues about ATM performance and behaviours.

As described, operational concept validation and concept development are two aspects of the process of changing concept ideas into reality that cannot be separated. An operational concept takes time to develop into an application and the validation process must allow for this ‘maturing’. The Concept Lifecycle Model aims to create a structure for the concept validation that also accounts for the concept development needs. It describes milestones in the development of a concept where fitness-for-purpose should be examined to avoid continued development without an indication of clear benefits or progress. An overview of the Concept Lifecycle Model was already presented in Figure 2-1; a more extensive description of the relevant phases is included in the first column of Table 2-2.

The ‘Concept Validation Methodology’ is most applicable to the phases V1, V2 and V3 of the Concept Lifecycle Model. V0 is considered as pre-requisite information for validation to commence. The later phases of Pre-operational (V4) and Operational (V5) are considered to be out of scope for R&D; different methodologies than E-OCVM are required for those phases. The Structured Planning Framework facilitates programme planning and transparency of the process. It consists of the following main steps, which are all further divided in sub-steps:




Table 2-1: High-level overview of steps of E-OCVM’s Structured Planning Framework

Structured Planning Framework Step Description 1. State concept and assumptions

2. Set validation strategy

3. Determine the exercise needs

4. Conduct the exercise

5. Determine the results

6. Disseminate information to stakeholders E-OCVM provides detailed descriptions of these steps and sub-steps, which intend as a support to identification of appropriate validation objectives based on an achievable validation strategy. Also, it explains the possible iterations between the steps and highlights that some steps may also be defined at a programme level. Finally, the case-based approach of E-OCVM serves to group information into common aspects in order to describe the potential of the concept under evaluation and thereby support the key stakeholders as they make the investment and implementation decisions. The main cases that should be anticipated are safety, human factors, business, environmental and technology. Each case will be developed along the concept lifecycle. When required a synthesis of the contents of these cases could be made to address the needs of identified stakeholder groups e.g., operational, safety regulators, ANSP investors, airline investors, ATM technology suppliers etc. 2.3. Fitting the parts together The three aspects described (Concept Lifecycle Model, Structured Planning and Case Based) fit together to form a process. This process is focused on developing a concept towards an application while demonstrating to key stakeholders how to achieve an end system that is fit for purpose. The Concept Lifecycle Model is the central aspect of the validation process. E-OCVM provides descriptions of how the Structured Planning Framework and the Case-Based Approach play a role in the phases V0 to V3 of the Concept Lifecycle Model. These descriptions are provided in Table 2-2. For the Case-Based Approach the description is high-level only; the current guidance document aims to provide more detailed information on the safety case from the Case-Based Approach.

Table 2-2: Use of Structured Planning Framework and Case-Based Approach in the phases of the

Concept Lifecycle Model (from [E-OCVM])

Phase and description Use of Structured Planning Framework and Case-Based Approach

V0: ATM needs The ATM performance needs and barriers must be identified. To complete the validation of the concept, the concept must show that it can alleviate these barriers enough thus enhancing ATM performance to the anticipated required level.

The information from V0 on performance needs and constraints is considered as being generated away from the R&D environment and is continuously updated by teams involved in monitoring ATM performance. It is essential pre-requisite information that the validation process will need in order to show how a concept addresses both a performance need and circumvents known constraints.

V1: Scope The concept should be described in

The Structured Planning Framework is used to plan the activities of V1 which will examine what activities will be required in order to evaluate the concept as it




Phase and description Use of Structured Planning Framework and Case-Based Approach

sufficient detail to enable identification of the potential benefits mechanism (i.e., the change to systems and/or operations that will enable a known barrier to be alleviated). Some aspects of the concept will be unknown or unclear at this stage. They may exist as a number of options to be assessed during the further validation process.

develops towards an application. The following steps are covered in V1: • Step 0 State Concept and Assumptions (including problem description); • Step 1 Set Validation Strategy (where evidence is needed to help to determine

strategy move to steps 2, 3, 4 and 5); • Steps 2, 3 and 4 may be used where exercises are needed to help determine

a suitable strategy e.g., fast time modelling activities to help identify the scale of the problem in different airspace or airports;

• Step 5 will collect any evidence from the exercises that will be used as input to the validation strategy

Cases: Supporting ‘cases’ will be created during this step. The cases will collect together stakeholder issues and will identify where evidence will be required to build the case – these needs for evidence will feed into the validation strategy.”

V2: Feasibility The phase to develop and explore the concept until it can be considered operationally feasible. During this phase system prototypes will be used that make assumptions about technical aspects in order to avoid system engineering which can be costly and lengthy. Aspects that should be focused on are operability and the acceptability of operational aspects. It is during this phase that operational procedures and requirements should become stable. The number of iterations depends on the complexity of the concept and how often unexplained situations occur that need to be explained. At the end of this phase HMI, Operating procedures (for normal and key non-normal conditions) and phraseology should be thoroughly tested. This stage will establish the behaviours of the new system.

Chosen concepts move to V2 where development will continue to a state whereby operational feasibility should be established. The Structured Planning Framework will be used to plan the validation exercises (continuously supporting the development activities). The following steps are covered in V2: • Step 0 State Concept and Assumptions (revisit previous work to ensure still

valid), • Step 1 Set Validation Strategy (revisit previous work to ensure stakeholder

issues captured, strategy still valid re platform capabilities and high level objectives clear)

• Steps 2, 3 and 4. Repeated for every exercise undertaken in V2. These are the main focus of the activities to establish feasibility.

• Step 5 Information for dissemination. Should be considered at any time during V2 once exercises start to generate results.

Cases: Appropriate information will be channelled through the cases. Additionally demonstrations will be used for showing feasibility and convincing stakeholders of fitness for purpose.

V3: Integration The phase to integrate any required functionality into pre-industrial prototypes. Engineering processes can be explored to provide experience that will be useful to building the endsystem. This phase is focused on integrating operating procedures by using realistic scenarios that are representative of what the concept must be able to manage in the target end-system. The focus is therefore on system level behaviour, performance and establishment of standards/regulations necessary to build and operate the required technical infrastructure. This work will enable costs and benefits to be clearly identified and provide information about the potential performance of the overall ATM system.

Feasible concepts move to V3 where integration takes place. The Structured Planning Framework will be used to plan validation exercises. The following steps are covered in V3: • Step 0 State Concept and Assumptions (again revisit previous work to ensure

still valid); • Step 1 Set Validation Strategy (again revisit previous work to ensure

stakeholder issues captured, strategy still valid re platform capabilities and high level objectives clear)

• Steps 2, 3 and 4. Repeated for every exercise undertaken in V3. • Step 5 Information for dissemination. Continuous during V3 once exercises

start to generate results Cases: These will be used to collect evidence about concept. If the concept appears to be successful these R&D cases will become the foundation for the cases in support of certification and implementation.

E-OCVM provides further information regarding key validation documents, validation information storage and dissemination, and support to validation.




3. Guidance for defining safety case activities Section 3.1 explains how the focus of safety case development in E-OCVM evolves with maturity. Section 3.2 provides high-level guidance regarding definitions, and Section 3.3 provides guidance regarding developing necessary expertise. Section 3.4 explains the use of E-OCVM in safety case development. Section 3.5 provides guidance regarding re-use of safety results. 3.1. Safety case development evolves with maturity The focus of development of the cases in E-OCVM evolves with increasing maturity; this also holds for the safety case. In general terms, in the early phases of E-OCVM safety case development aims at collecting sufficient evidence for decision-makers to decide whether a concept will be implemented, further developed, improved, or not further developed. While moving towards a more consolidated concept, as represented in Figure 3-1, the emphasis changes: • From providing early feedback for major design improvement and preliminary evidence that the

concept could be safe, • To collecting sufficient information that the concept as finally developed and to be implemented, is

indeed acceptably safe. This means that the underlying concept is safe, that the design is complete, that the system is safe under all conditions, and that the design is robust against abnormalities.

Figure 3-1: Focus of safety assessment during the concept lifecycle.

In Section 4.3 detailed guidance will be given on the objective of the safety case per phase of E-OCVM. 3.2. Guidance regarding definitions Care is to be taken in using definitions. For several relevant terms, multiple definitions exist. It is of importance to determine and use the best applicable definitions. It is equally important to clearly describe in safety case documentation which definitions have been used. [SRC DRAHG] provides an overview of different existing definitions, and proposes how to deal with these differences. An example is given for a term not considered in [SRC DRAHG]: Several definitions exist for ATM, which do not all agree and which are not all fully clear on whether and which airborne functions are included in ATM. For safety it is important to analyze well the full operation including ground-based and airborne parts. The Single European Sky legislation Regulation (EC) No 549/2004 [EC 549/2004] lays down the framework for the creation of the Single European Sky, and is very explicit on this. It is recommended here to adopt this definition: ATM is “the aggregation of the airborne and ground-based functions (air traffic services, airspace management and air traffic flow management) required to ensure the safe and efficient movement of aircraft during all phases of operations”. In terms of an airline, this ATM definition covers air transportation safety rather than ATC.

Improve the concept

Prove the concept is good enoughHF Body ofEvidence

More matureconcept/system

Initialconcept(s)

Human Factors processes

Safety Assessment

V1 V3

Evaluate if the concept is good enough Body of evidence

Advancing maturity




3.3. Guidance for developing expertise As explained in the purpose of this document (Section 1.2), several emerging needs exist for safety case development of large, SESAR-like changes, and several new approaches, complementary to approaches already in use, are emerging that aim to address the SESAR-identified emerging needs. A weak link in the application of such emerging approaches in safety case development in R&D is the basis of expertise. Therefore there are complementary needs of advanced safety courses and hands-on safety learning. These safety courses should provide a broad view on approaches currently in use for safety case development, on emerging needs, and on emerging advanced approaches, as discussed in this guidance document. The application of emerging approaches should develop hands-on experience and also lead to their improvement. 3.4. Guidance for using E-OCVM This section provides high-level guidance for safety case development in line with E-OCVM. The following basic steps are recommended: A. Select the phase of E-OCVM’s Concept Lifecycle Model to be tackled. B. Determine objective and scope of safety analysis in line with the selected phase. C. Determine methods and techniques to be used. D. Document the results in a safety case. Below, for each of these steps further guidance is given. A. Select the phase of E-OCVM’s Concept Lifecycle Model to be tackled.

For selecting the phase of E-OCVM’s Concept Lifecycle Model to be tackled, the following sub-steps can be defined:

i) Determine which previous safety analysis results are available for the considered concept. • Eurocontrol’s Validation Data Repository (VDR; see [VDR]) is available for retrieving

(and storing) validation results. • ‘Negative’ validation results are of high importance in learning why concept versions

that were already assessed were not sufficiently safe. • Section 3.5 provides further guidance on re-use of previous safety analysis results.

ii) Determine the maturity of the concept according to the Concept Lifecycle Model. • Section 5.9 introduces the concept of ‘transition criteria’ that can be used to evaluate

the maturity of a concept under consideration.

iii) Review the existing safety analysis results, and determine for which phases of the Concept Lifecycle Model the E-OCVM objectives of safety case development have indeed been reached.

• It is recommended to consider to which extent the existing safety analysis results have addressed the emerging needs in safety analysis of major, SESAR-like changes discussed in [CAATS II D13].

• Section 5.9 includes a discussion of ‘safety case specific’ transition criteria, developed specifically for evaluating the maturity of a concept under consideration from a safety case point-of-view.

• So far, few safety case development activities have worked following E-OCVM. This means that few sources explicitly state which phase they have covered.

iv) Select the phase of E-OCVM to be tackled.




• If safety case development runs in pace with the concept maturity, and the objectives of previous phases have indeed been reached according to step iii), then the selection is straightforward: the concept maturity will determine the phase to be tackled.

• If safety case development does not run in pace with the concept maturity, then the situation is more complicated. In case safety case development is lagging behind, then the project runs a risk of discovering safety problems in a late stage, which may lead to costly redevelopment, or even potential no-goes. In case safety case development is running up front, then similar project risks may arise from other development and/ or validation aspects.

B. Determine objective and scope of safety analysis in line with the selected phase.

• The general objective of the selected phase can be determined from E-OCVM (cf. Table 2-2). The focus of safety case development changes with maturity (cf. Section 3.1), and hence the desired type of feedback from the safety case may change with maturity.

• In early phases, the desired type of feedback from the safety case may be directed towards the developers. Doing several iterations between concept development and risk assessment feedback is an effective way of reaching the objective of a phase. The iterations are as follows: Concept developers deliver an improved and/ or more detailed version of the operational concept(s). The safety validation activities provide feedback to the concept(s), which the developers can next use to improve their concept(s).

• The safety regulatory framework is of importance for determining objective and scope. Section 5.3 provides guidance for addressing safety regulations.

• In scope definition, one should start from a wide ATM scope, including airborne and ground-based elements, before addressing further scoping. It should be realized that the smaller a scope is defined, the more difficult it is to identify risks emerging from the dependencies and interactions between different elements.

• Further care should be taken in adopting the best applicable definitions (cf. Section 3.2). • In Section 5 it is discussed how the SESAR-identified emerging needs in safety case development

of large, SESAR-like changes can be addressed. This is also of importance for definition of scope and objectives. As an example, objective and scope should be co-ordinated with the other cases; guidance regarding defining relations with other cases is provided in Section 5.10.

C. Determine methods and techniques to be used.

Tailor the approach to be used towards the objectives and scope. • Section 4 provides guidance on using safety analysis in ATM R&D to reach E-OCVM objectives,

including guidance on selection of methods. • Section 5 discusses how the SESAR-identified emerging needs in safety case development of

large, SESAR-like changes can be addressed. • Co-ordinate the planned safety case development activities with other validation activities. If the

teams working on different validation aspects work in a parallel and coherent way, duplication of activities may be prevented, and use can be made of complementary views. Section 5.10 provides guidance on the relations of the safety case with other cases.

D. Document the results in a safety case.

• Documenting explicitly which phase is tackled in the safety analysis, helps both the project itself and the future use of the results.

• Conclusions of the safety case should be supported by convincing arguments. It should be traceable how these arguments are supported by evidences.




• Where applicable, stating that the system has a negative impact, or that there still remains a high level of risk, is as valid as stating where a system has a positive impact. Typically in ATM R&D it is very much of interest to know why a certain concept is not safe enough; such knowledge is vital in R&D to enable operational concept developers to improve the concept.

• Eurocontrol’s Validation Data Repository (VDR; see [VDR]) is available for storing (and retrieving) validation results.

3.5. Guidance for re-use of safety results To re-use the results of safety assessment across multiple R&D projects investigating the same concept would mean to reduce the cost of safety analysis and to join the analysis capabilities of different teams with an evident advantage in terms of effectiveness and efficacy. However this is an issue that received limited attention and research effort till now. There are neither methods considering specifically the re-use of previous results, nor specific techniques and tools facilitating it. A set of conditions are needed to facilitate the re-use of safety results across different projects. These conditions are listed in the following. • Formal identification of the system under analysis and of the operational conditions considered. If the

extent of the safety analysis is not well defined potential future users of the analysis results will not be in condition to understand what has been included. Possible examples of questions concerning the identification of the system are: is the software considered as part of the system and are its potential failures included in the analysis? Is the concerned aerial part (and the possible pilot behaviour) considered?

• Definition of the assumptions and of their implications. Assumptions are an essential and unavoidable element of safety analysis. They are often necessary to provide a frame for the evaluation process, but, they can also have a powerful effect on the conclusions of the safety analysis that should not be underestimated. The introduction of unmotivated assumptions in safety analysis is a widely recognised issue in the scientific literature [Tversky & Kahneman, 1974]. Different projects tend to have different assumptions, and often the implications of those assumptions are not adequately investigated. A clear definition of the assumptions adopted and an analysis of their implication are essentials to allow re-usability of results.

• Standardised assessment methods. A standardised assessment method would facilitate a better understanding of the extent and type of analysis done to experts not directly involved in the assessment activities. In addition, a standardised assessment method allowing to partition the analysis, would allow a more direct and substantial re-use of the analysis of those system/concept components that are not changed during the concept evolution.

• Use of standard templates. The adoption of standard templates would facilitate the identification of the issues of importance in a usually wide, and not easy to browse, software analysis documentation, especially with regard to lessons learnt. This would facilitate communication and mutual, easy, understanding.

• Public repositories of safety assessment results. Public repositories would be needed to store safety assessment results in a standardised format, and to allow an easy identification of the issues of relevance.

In [CAATS II D13] it is discussed what is available or under development to help safety analysts in satisfying each of conditions presented above.




4. Guidance for using safety analysis Any validation process is to be supported by analysis; this also holds for safety validation. In [AP15 Toolbox] the main activities to be part of any safety analysis are described, as agreed upon by FAA, Eurocontrol, and other involved organizations in FAA/ Eurocontrol Action plan 15: ATM Safety Techniques and Toolbox [AP15 Toolbox]. It is “the first major attempt to evolve a common inter-operable safety approach”. Section 4.1 provides an overview of this safety analysis process, as copied from [AP15 toolbox]. Section 4.2 provides details on the stages, as also copied from [AP15 toolbox]. In Section 4.3 it is clarified which safety analysis stages are needed per phase of E-OCVM’s Concept Lifecycle Model. Finally, Section 4.4 provides guidance on the use of individual methods. 4.1. Overview of a safety analysis process Safety assessment methodology is usually focused on ensuring that new proposed changes do not increase risk from a safety perspective. This means that all possible impacts of a new operation or system should be assessed, and their combined risks determined. These potential impacts can be intended (e.g. reducing separation minima, and therefore bringing aircraft closer together), or unintended (e.g. introducing data-link technology, which can have indirect safety impacts such as reducing the risk of call-sign confusions, but possibly introducing new errors such as up-linking messages to the wrong aircraft). Initially, a safety assessment considers the proposed operation or system definition (often called the Operational Concept), and analyzes how it could impact matters, for the better and/or for worse, with respect to safety. This analysis involves considering the scope of the assessment (affecting how far the analysis is taken particularly in terms of interactions with other system elements), and then identifying all possible hazards and the severity of their consequences. The analyst then determines how probable these failures are, as well as how likely the system is to recover form such failures. This culminates in an overall risk estimate for the system. Usually at this point, this risk or safety assessment must be compared to a benchmark, such as existing system risk to see if it is an improvement or not. It is here that a ‘Target Level of Safety’ is often used. This will express for example, the tolerable (to society) frequency of an accident, in terms such as accidents per flight hour, or per approach/landing, or per surface movement. The TLS allows decision-making on whether or not to continue developing the concept, or to continue but with key safety requirements that need to be demonstrated in the new system for it to be adequately safe. Once such a safety process is conducted, it is documented as a ‘safety case’, and used to justify to the regulatory authorities that the new proposed system or system change will not adversely affect safety. However, because the safety case will often contain safety requirements and assumptions that are key to ensuring that the system remains within its safe operational envelope, it should be seen as a living document, and be periodically updated. Ideally it contains information that is utilized initially by the system designers and then by the operations people for the remainder of the system’s lifecycle. Once the new design itself is operational, there becomes a need to continually monitor safety performance so the responsibility for safety oversight then transfers to the management of the operational facility. Usually a safety activity will be created that will record safety-related events (e.g. loss of separation, TCAS events, etc.), for lessons learned purposes. Trends may occur for example related to local factors (e.g. particular controller working practices and changes in local sector design) or more widespread factors (e.g. shifts in controller demography and availability). The detection of trends that could compromise safety requires archiving the relevant data and monitoring them continuously. The process cannot rely on human memory. When such a trend is detected and determined to be operationally significant, an




appropriate reaction should occur to ensure that the system returns to its optimal safe performance. This amounts to safety or organizational learning (see the final ‘step’ in this section). This is still part of the system safety process, and indeed such information on the causes and contributors to incidents and accidents needs to be fed back to safety assessment practitioners, enabling them to refine their tools and techniques. The challenge to proactive management of safety is discovering the precursors of the next accident, identifying their causal factors, and implementing the most effective interventions before an accident occurs. Safety Assessment of an air traffic operation can therefore be seen as a seven-stage process, as shown below (with feedback leading to Organizational learning as a potential ‘eighth’ step that could be developed for the industry).

Figure 4-1: The generalised Seven-Stage Safety Assessment Process, plus a key eighth step of

organizational learning (from [AP15 Toolbox]).

The following subsection outlines the key aspects of these seven steps, plus a key eighth step of organizational learning, and begin to identify what techniques can be used at each stage (several techniques can be useful in more than one stage (...). 4.2. Detailed description of the stages of the safe ty analysis process The following paragraphs outline the key aspects of the stages of the safety analysis process, including the ‘eighth’ stage dedicated to feedback provision.




Stage 1 - Scoping the Assessment This stage for a development project entails the availability of an Operational Concept or System Specification. It is difficult to conduct system safety assessments without knowing the system operational concept. However, it is not uncommon in early assessments that the operational concept itself is a living document and ‘ever-evolving’. The Project and the safety practitioner must develop a Safety Plan that specifies the scope of the safety assessment and outlines the approach. This can include such pertinent information as what Target Level of Safety (TLS) (or part of it) is relevant for the safety assessment, where the system boundaries are considered to be, and the relative focus on aspect such as hardware, software, and human elements of safety. This helps the safety assessor determine at an early stage the likely techniques to be used, and helps the Project Manager envision the likely safety-related resources such as access to operational personnel, the need for simulations and trials, etc. The Scoping stage is therefore partly technical (identifying the likely characteristics of the safety assessment based on an initial assessment of the nature of the proposed change), and partly administrative and regulatory. Nevertheless, the importance of the administrative/regulations component should not be under-estimated. When a hazard arises in an existing system, the scoping of the required assessment will vary considerably depending on local factors and company procedures. Nevertheless, there will still be a need to consider the nature of the hazard, and this will depend critically on the tool or technique to be used. Many hazards that arise in existing systems may be Human Factors-related, yet tools for recording incidents, etc. often record too superficially the information required to scope a study (see however HERA-JANUS in the Toolbox section). Therefore, safety issues arising in existing Operational Units, often will require an initial scoping investigation; talking with operational personnel to better understand the issue. The TLS may still be used, but more often if it is a local issue, safety assessment and interventions may be more qualitative in approach. They may for example identify the hazard and move straight to developing mitigation measures, after a qualitative assessment of the risk. Outputs: Safety plan; assignment of safety/risk criteria (e.g., TLS) Techniques: Scoping does not always use defined techniques, and may be informed by assessor judgment and incident/accident experience, and prior practice in a related area; the TOPAZ accident risk assessment methodology may also be used. The approach will depend on local adaptation and the organization’s Safety Management System (SMS) The FAST methodology helps scope the assessment by defining Areas of Change in the Concept of Operation. Stage 2 - Learning the nominal operation Safety Assessment is ‘transitive’ in nature – it requires an object, something to analyze. This is often not realized by non-safety practitioners. There is therefore a need to learn about the description of the operation and systems as it should work or function; this being the nominal ‘model’ (how the system should behave), from which the ‘risk’ model (how it can fail, and how it can be ‘recovered’) can be developed during and after the hazard identification phase. There are various ways of modeling an operation for subsequent safety analysis, and indeed often this is done by the Project or Program in any case. Examples are Functional Block Diagrams or Use Case Modeling. In some cases, special modeling approaches might be required such as Task Analysis for modeling human interactions. Some of these are considered in the Toolbox section. These techniques are effectively abstractions of the system from a particular viewpoint, and so the exact safety modeling requirements are a function of the aspects on which the safety practitioner intends to focus on. For existing systems, paradoxically, there may be no abstraction of the system available, particularly for the human (i.e. controller) tasks. Nevertheless, the safety practitioner will usually find it necessary to construct a representation of the system to properly assess it, and so techniques such as task analysis




can be used for such purposes. The advantage for the safety assessor with existing systems is often that observation is feasible, and fewer assumptions have to be made, since the assessor can simply interview controllers or pilots or other operational experts. Outputs: Description of operations and systems used. Techniques: Hierarchical Task Analysis, TOPAZ accident risk assessment methodology, and SADT. Additionally a number of other system modeling techniques exist, but these vary in usage in ATM, and ATM is in fact still exploring best techniques to use. This area will therefore be redressed in later versions of this report. Stage 3 - Identifying hazards Probably the most critical stage in safety assessment is hazard identification and risk assessment. Such risks include those that may emanate from the Operational Concept itself; e.g., related to proposed hardware, software, procedures, and/or human elements. These may relate to ‘external events’ in the environment (e.g., bad weather), or to failures or events in other systems that can affect the system under consideration. One of the difficulties of hazard identification in ATM applications is that it is effectively a globallyinteroperable system. This means firstly that it is difficult to know when a hazard identification exercise is complete. Secondly, it means that there is much to consider, especially in terms of interactions of system elements. Certain failures (e.g., power supply) will affect multiple systems, and loss of key data similarly can affect different systems in different (and sometimes unexpected) ways. These are called common cause failures (identified by Common Cause Analysis), and relate to what are called ‘dependencies’ between systems, and can lead either to new failure outcomes or elevated failure frequencies, so they need to be identified. Most hazard identification techniques fall into two categories, namely single-assessor and group-based approaches. The single-assessor approach usually entails rigorous analysis of all aspects of a system according to a failure schedule or list of failure types. Some techniques are specifically aimed at certain hazard types (e.g., human error) whilst others are generic across different hazard categories. The group-based hazard identification approach involves doing this with a group of experts rather one or two assessors. The main challenge for both approaches involves shifting the boundary between imaginable hazards and unimaginable hazards. In addition to hazard identification by experts, there is the option to use recorded observations, either from actual operations (e.g., using databases such as ASRS or radar track data) or from real-time and non real-time simulations. The former should ideally always be consulted when conducting hazard identification, to see if past experience can offer information about likely hazards and hazard interactions. The latter (real-time simulations and/or non-real-time simulations incorporating human performance models) can similarly be used to identify hazards in operability of a system, and can gather insights about potential errors that could contribute to hazards. They can also of course identify ways to mitigate or control hazards. In current operations and systems, hazard identification is sometimes the starting point, since a series of hazard-related incidents may have occurred due to certain causes. The safety practitioner’s job is then to investigate these incidents to find the complete set of causes, as well as possible alternative hazards that could arise, and derive mitigations to reduce incident rate or severity. Although such investigations will not usually follow a formal safety assessment pathway, some of the techniques can still be helpful to ensure that the specialist or practitioner has a complete understanding of the hazards, risks, causes, and contributory factors. Output: Defined hazard set




Techniques: Air Safety Database, ASRS, Common Cause Analysis; External Events Analysis; FAST; FMECA; HAZOP; Human Factors Case; TOPAZ accident risk assessment methodology; TRACER-Lite, PDARS Stage 4 - Combining hazards This stage means developing a way to aggregate the different identified hazards and their contributions to accident sequences into a risk model with which the total risk due to the proposed system or change can be evaluated. This stage is necessary in all but simple systems or narrowly-scoped analyses, because otherwise it becomes difficult to weigh up the different identified risks and their various accident sequences, and in particular to determine if the risks will be within the Target Level of Safety selected. Typically, at the top level of a risk model there is a logic diagram such as a fault or event tree, which models respectively the causes of an event (usually a specific hazard), the resultant consequential pathways after an event and a collision risk model at the end of the pathways. These logic diagrams define according to strict rules how events can link together to cause a hazard, and how such hazards can either propagate to accidental consequences (such as mid-air collision, runway incursion or Controlled Flight into Terrain), or else safe states (via mitigations or safety nets). Such ‘trees’ can become quite complex, and usually they are analyzed by specially-designed computer tools. Since levels of risk are influenced (possibly quite significantly) by dependencies and common cause failures that exist between different parts of the risk model, risk modeling should include a dependency analysis (e.g. going through the risk model identifying common elements and dependencies in particular concentrating on ‘AND’ gates if using fault trees, for example). A complementary approach is to make use of a Monte Carlo simulation model which allows to evaluate multiple dynamical and dependent events, ‘non-nominal’ scenarios, and permutations of such events and scenarios, and to make effective use of a larger variety of qualitative and quantitative input data (e.g. human performance models). Such an approach is also more powerful in providing insight in the effectiveness and sensitivities of the interplay between multiple humans and systems involved in the operation (e.g. controller and pilot, aircraft systems, ATC system). If properly applied, it can make assumptions explicit and with this make the Monte Carlo simulation results open to scrutiny by operational experts. The result is a risk model that encapsulates and relates the different hazardous and recovery events into a homogeneous model. This risk model can then be quantified (this process is called ‘evaluation’), delivering not only the overall risk estimate, but also the ability to determine which elements in the operation are most safety critical. This then in turn points the way towards risk mitigation. The risk modeling is therefore one of the most critical parts of the overall safety assessment process. Output: Risk Model Techniques: Bow-Tie; Collision Risk Models; Common Cause Analysis; Event trees; Fault trees; Human Performance Simulation; TOPAZ accident risk assessment methodology. Stage 5 - Evaluating Risk Having developed a risk model that is logic-based and/or simulation-based, the next stage is to determine the quantitative properties of the risk model – in particular how often the various events are likely to occur. In some cases, databases will exist which can give such information, e.g. the likely time before failure of a radar screen, or the probability of a communication error between controller and pilot. In other cases, there may be techniques to estimate such values.




When failure data are collected for a component, or for a particular human task, there will always be some uncertainty in the data derived, due to limits on data samples, and due to slight performance differences between the same components, and rather large potential differences between individual human 'components’. Therefore, having amassed the data required to ‘evaluate’ the risk model, considerations of residual bias in the data-set and uncertainties, and how they can interact, should occur. This requires expertise, but represents good practice for safety assessments. In particular, if there are too many uncertainties in the data, then comparison against a quantified Target Level of Safety will be unreliable. For those parts of the tree where a simulation model has been developed, large scale Monte Carlo simulations and sensitivity analyses are performed and documented. In addition, a formal bias and uncertainty assessment method can be applied. Where no databases and no appropriate techniques exist, there can be recourse to expert judgment, using formal procedures and validated experts. However, because expertise is known to suffer from biases, and since by definition experts on the failure behavior of future systems have limited expertise, expert judgment protocols must include means for detecting biases and incoherent judgment, and hence rejecting the results should the expertise fail according to certain quality criteria. On those places in the tree for which Monte Carlo simulations have been performed, it is also possible to compare the results of the simulation model with the experts judgment and, in case of differences, to discuss this with the experts. This often will lead both to better expert judgments and to a better simulation model. The quantification of risk is unfortunately sometimes seen as a ‘numbers game’, relying on questionable data, crude modeling of scenarios and subsequent simplistic mathematical treatment. However, it is relevant to point out that a number of accidents have been predicted beforehand, but ‘without the right numbers’, hence underestimating their risk, and thus remaining unprepared for the accident. This could be seen as reinforcement of the position against quantification, however this would be short-sighted. Most accidents are complex and involve both related and unrelated factors and events, difficult to predict outside of complex risk modeling. Without quantification, such accidents tend to be assumed to be rare or negligible (due to a natural human bias called 'conservatism’). Therefore, the drive instead should be to derive better numbers by collecting and sharing event and incident data, so that when accident sequences are identified, their likelihood is accurately predicted. Output: Evaluated Risk Model; identify and evaluate dependencies, evaluation of risk against target criteria; risk-informed decision-making becomes possible Techniques: ASRS; Human Error Database; Bias and Uncertainty Assessment; Collision Risk Models; Common Cause Analysis, FAST; TOPAZ accident risk assessment methodology; HEART. Stage 6 - Identifying potential mitigating measures This stage involves four main steps. The first step is to consider whether risk reduction is required, i.e. whether the safety target criteria are met. This sets the initial obligation to reduce risk, and tells the assessor the size of the challenge ahead (particularly if the target level is not met). If the risk is in the broadly acceptable area the risk level is such that effort to achieve further reductions is likely to be grossly disproportionate (although the duty holder is still expected to demonstrate this). If the risk is seen as being ‘tolerable’, no risk can be accepted unless reduced as far as reasonably practicable. Therefore this first step is concerned with what must be done, and then what should be done to reduce risk and increase safety. The second step is to determine where the major element of risk is coming from, i.e. what part of the risk model is contributing most risk. This is the natural target for reduction. Some techniques (e.g. Fault Trees)




can automatically generate a prioritized list of the events in terms of their contribution to risk. The third step is then to support design developers in identifying potential mitigations or changes that could reduce risk. Sometimes the major element of risk cannot be mitigated, so other lesser elements must be tackled, which together would lead to the required risk reduction. The fourth step is then to re-calculate risk, having adjusted either the model or the quantitative inputs according to the mitigations developed (called safety requirements) to verify that the system is acceptably safe. A word of caution here is that it is easy to over-estimate the impact of reduction measures and mitigations, and also it is easy to overlook unforeseen interactions and problems associated with the mitigations themselves. In fact in several industries after the identification of reduction measures or mitigations, it is a requirement to do further hazard identification to detect such unplanned interactions, followed by requantification of the risk model. Another aspect of this stage concerns tracking safety requirements and assumptions. For design and development projects, these may occur over a long timescale. This means that either there will be several safety cases during the development life cycle, each becoming more detailed as design detail increases, or else the safety case work may be more of a continual and iterative process, gradually leading to a definitive safety case. In either of these situations, there is a need to track the key safety assumptions and requirements as they are made by the project, and ensure that they are enacted in the actual design of the system. This may mean that there are key training and procedural assumptions, or requirements concerning the Human Machine Interface (HMI), or key performance requirements of equipment that need to be assured and tested during equipment or system performance simulations or trials. Furthermore, the designers and developers may realize later in the process that they wish to change certain design parameters, and will want to know the impact on risk. A mechanism for enabling the impact of such changes to be rapidly seen is therefore desirable. Such a hazard and requirements tracking and impact evaluation technique has indeed been developed, and can therefore be used to keep track of all requirements, make sure they happen effectively, yet to allow some flexibility so that safety is not seen as a designers’ ‘strait-jacket’. Output: Potential mitigating measures to reduce risk Techniques: HAZOP; Human Factors Case; TRACER-Lite; HTRR, Bow-Tie, TOPAZ accident risk assessment methodology. Stage 7 – Safety monitoring and verification With respect to the existing systems, this stage refers to the need to continually monitor overall system safety performance and determine if the various safety requirements are performing their functions as expected. It requires a means of monitoring and analyzing resultant safety data, and then drawing lessons from those data in sufficient time to react and prevent accidents from occurring. This is not trivial, and requires pre-definition of safety parameters and events, automatic and manual recording mechanisms, analysis tools, and data storage and retrieval systems (knowledge bases). It also of course requires a good safety culture that will accept such monitoring and analysis and will act on its conclusions, and a legal framework (a so-called ‘just culture’) that will protect controllers and pilots offering up much-needed safety information on human errors and other events that occur. Actual collection, analysis and sharing of safety-related data allows the whole safety framework to become a learning system, leading to better safety data and safety evaluation techniques, and safer systems. Expectations and a system for monitoring should also be established prior to implementing an intervention in order to measure its effectiveness and identify any unexpected effects from system operations. Output: Measurement of safety-related events & data against predictions Techniques: ASRS, PDARS, Air Safety Database, FOQA, FDM (...)




Stage 8 – Feedback to Operation, Assessment, and De sign In Figure 4-1 there are two feedback loops – the first refers to ‘Iteration’, meaning that safety assessment is usually iterative in nature and safety assessments themselves are not always ‘once-through’ processes. The right-hand feedback loop however refers to feedback at a more organizational level, involving three key parties. The first is clearly Operations, in that hazard and risk information can be of use to actual Operational Centers in their own safety management practices (including safety-related training for controllers). This may be of particular relevance when for example an assessment for a project uncovers new hazards that may apply to other projects or even existing systems. The second party that can benefit from structured feedback are safety assessors themselves, since then assessors working on new system assessments can see what hazards etc. were identified, with what risk levels, and with what mitigations. Assessors need not be constrained by prior assessments, but should be able to view them. Therefore a ‘library’ of safety assessments can be useful in this respect. The third party that can benefit from feedback are designers and developers of new concepts. Such people are not necessarily habitual readers of safety assessments, and yet if such information could be presented in a usable way to designers/developers, then they would be considering safety aspects from a very early stage in their concept formulation processes. Safety assessment practice is therefore a potential source of organizational learning for the industry, which could enhance Safety Management efficiency and effectiveness. This step has yet to be properly developed for ATM, but is a logical addition to the ATM safety management approach. A critical component of safety is the tracking and analysis of safety data to enhance awareness of potential hazardous situations. The collection, analysis, and sharing of safety data supports the continual improvement of safety in ATM. Various techniques exist to collect, prepare, and analyze data (quantitative and textual) to support feedback of information to stakeholders. 4.3. Safety analysis stages needed per phase of E-O CVM This section provides guidance on what needs to be done in terms of safety analysis to satisfy the requirements that E-OCVM poses per phase to inform stakeholders. This guidance is presented in Table 4-1. For each phase, it is described which stages of safety analysis need minimally be done to satisfy the demands posed by E-OCVM for that phase. These descriptions logically follow from the E-OCVM descriptions of the phases of the Concept Lifecycle Model, and of how the Structured Planning Framework and the Case-Based Approach fit on the phases (as copied in Table 2-2).

Table 4-1: Safety analysis process stages needed to satisfy the demands posed by E-OCVM, as

identified from analysis of Table 2-2

E-OCVM phase Safety analysis process stages needed V0: ATM needs Part of stage ‘Scoping the assessment’ involves identification of ATM performance

needs with respect to safety (e.g., safety targets). Part of stage ‘Modelling the nominal system’ can be used to analyze well the existing operations, to support identification of the ATM barriers that need to be alleviated to reach the ATM need. Accordingly, parts of stages ‘Scoping the assessment’ and ‘Modelling the nominal system’ need to be done to satisfy V0.

V1: Scope

The stages ‘Scoping the assessment’ and ‘Modelling the nominal system’ involve identifying and learning the target operational concept and setting up the validation strategy with respect to safety. Accordingly, AP15 stages ‘Scoping the assessment’ and ‘Modelling the nominal system’ need to be done to satisfy V0.




E-OCVM phase Safety analysis process stages needed Furthermore, to determine an optimal validation strategy that is based on evidence, one or more full cycles of the seven plus one stages of the safety analysis process may need to be completed (as follows from E-OCVM statement on steps 2, 3, 4, and 5).

V2: Feasibility To refine the concept and to eventually provide evidence that it is sufficiently safe, one or more full cycles of the seven plus one stages of the safety analysis process need to be completed. Results may be fed back after completion of a full cycle of the safety analysis process, but also after completion of a subset of the stages.

V3: Integration To provide evidence that the further detailed operational concept is sufficiently safe, one or more full cycles of the seven plus one stages of the safety analysis process need to be completed. Results may be fed back after completion of a full cycle of the safety analysis process, but also after completion of a subset of the stages.

4.4. Guidance for the selection and further develop ment of safety techniques The optimal safety case development approach in a phase depends further on the question at hand; e.g., the number and type of stakeholders involved, the safety target and scope selected, the applicable safety regulatory framework, and the type of change which the concept proposes. Safety assessment methodologies as SAM [SAM], ED78A [ED78A], TOPAZ [TOPAZ] and SAME [SAME PT 1], allow the use of a wealth of techniques for the individual safety analysis stages, which may be selected depending on the question at hand. The following table presents an overview of established techniques and advanced needs for supporting techniques that have been identified by AP15 for use in SESAR and its USA counterpart NextGen by 2012 onwards. The advanced needs have been identified per stage of the safety analysis process of [AP15 toolbox], and for several needs, candidate advanced emerging techniques have been identified. It should be realized here that these advanced emerging techniques need their own maturing process, which could be considered as a lifecycle starting from the identification of the need for the technique, running through e.g., its development, testing, documentation, evaluation, and maintenance of the technique. Therefore the maturity of each technique is indicated, in terms of three status levels: • Started = Development has started • Proposal = An already developed approach has potential • Applied = An approach has demonstrated to work For further information on the techniques included in the table, one is referred to [Safety Methods Database].




Table 4-2: Overview of needs for new supporting techniques for use by SESAR and NextGen by 2012

onwards as from [AP15 ppt]. Per stage of the analysis process of [AP15 toolbox], established

techniques are presented in the second column. In the third column the advanced needs are

presented, where possible including candidate advanced techniques. The final column indicates the

status of maturity of the candidate advanced technique.

Stage Established Advanced Status

Stage 1 – Scope the assessment

• ANSP directed • Failure directed • Current TLS • Current RCS, e.g. ED125 • Current Risk Picture, e.g.

IRP & CATS • Current legal framework

• Multi-stakeholder directed • Success included • Future TLS, e.g. Episode3 • Future RCS, e.g. SRU • Predicted Risk picture

e.g. Predictive IRP/STAR • Future legal framework

Proposal Applied Proposal Proposal Proposal Started

Stage 2 – Learn the nominal operation

• Functional Modelling e.g. SADT

• BADA aircraft models • Human In The Loop

(HITL) simulation • Task Analysis

• Goal directed modelling e.g. I* analysis

• Realistic Flight Sim. (RFS)

• Human workload simulation e.g. Air-Midas

Started Applied Applied

Stage 3 – Identify hazards

• Functional Hazard Ident. • Safety & Hazard Data

Bases e.g. Tracer-lite • HAZOP • FMECA • Common Cause Analysis • External Event Analysis

• Human Factor Issues Ident.

• Pushing the Imaginable e.g. FAST

• Safety Oriented HITL e.g. SAFSIM

• Human model & RFS e.g. Air-Midas & RFS

• Contextual issues ident. • Organizational issues

ident.

Applied Applied Applied Applied Started Started

Stage 4 – Combine hazards

• Swiss cheese model • Fault Tree Analysis • Event Tree Analysis • Human Reliability

Analysis, e.g. HESRA • Blunder risk modelling • TCAS model, e.g. InCAS • Collision Risk Models,

e.g. by ICAO

• Conflict Scenarios • Human performance • Success approach • Multi-Agent Modelling • Organizational Modelling • Stochastic analysis

e.g. by TOPAZ

Applied Applied Applied Applied Started Applied




Stage Established Advanced Status

Stage 5 – Evaluate safety risk

• Expert based estimation • Statistical Data Collection • Data in publications

e.g. HEART • CRM calculations

e.g. ICAO accepted • Point Estimated Risk

• Monte Carlo simulation • Calibration Reference

Data • ATCo/Pilot reliability

e.g. CARA • Collision Risk Simulation

e.g. TOPAZ • Bias & Uncertainty

Analysis e.g. TOPAZ

• Mitigate Proprietary Issues

Applied Proposal Proposal Applied Applied ?

Stage 6 – Support safety risk mitigation

• Expert based mitigation • HAZOP • Barrier Model • Dependability reqmts. • HERA

• Resilience Engineering • Software Assurance

e.g. SWAL • Procedure Assurance

e.g. PAL • Human Assurance

e.g. HAL

Proposal Applied Proposal Started

Stage 7 – Safety monitoring and verification

• Hazard Tracking in SMS • Safety Reporting Systems • Flight Reconstruction

e.g. PDARS • Dedicated HITL

simulations • Dedicated Flight Trials • Normal Flight Operation

e.g. LOSA

• Handling uncertainty • Multi-source data

e.g. ASIAS • Normal ATC Operation

e.g. NOSS, Day-to-day

? Proposal Applied

Stage 8 – Feedback to operation, assessment, and design

• Validation Data Repository

• Risk as a function of minimum separation value (e.g. ICAO)

• Safety Cases availability • Transition to next stage in

E-OCVM, e.g. SARD, CAATSII

• Relations with other cases, e.g. CAATSII

• Sensitivity analysis, e.g. TOPAZ

• Identify concept bottlenecks, e.g. TOPAZ

? Proposal Proposal Applied Applied

It is recommended to gain experience with the advanced techniques, both aiding projects to take benefits of these new methods, and aiding evaluation and further development of these methods, such that the needs can be addressed.




5. Guidance for addressing SESAR-identified emergin g needs 5.1. Introduction This section presents guidance for tackling the SESAR-identified needs that are emerging for safety case development in R&D for advanced developments. This guidance is based on the following results from the analysis of sources documented in [CAATS II D13]: • an overview of SESAR-identified emerging needs; • a discussion of established and emerging approaches that support addressing these needs; and • the observation that integration of the various novel emerging approaches with each other and with

established approaches typically has not yet received sufficient attention. Table 5-1 provides an overview of the SESAR-identified emerging needs, as copied from [CAATS II D13]:

Table 5-1: Overview of SESAR-identified emerging needs for safety case development in R&D for

advanced developments

Id Description of emerging need A. The need for a ‘macro’ safety case

B. The need to address safety regulations

C. The need to address the multi-stakeholder nature of advancing air traffic operations

D. The need to address the success side of a change

E. The need to cover the human operators in the ATM system

F. The need to identify unknown ‘emergent’ risks

G. The need to address E-OCVM requirements

H. The need to assess concept maturity

I. The need for managing relations between cases For each of these emerging needs guidance is given in a dedicated subsection. Each subsection starts with an explanation of the emerging need (as copied from [CAATS II D13]). Then, a description of the approaches that aim to address the emerging need is presented (as from [CAATS II D13]). Each subsection ends with guidance on how these approaches may be integrated. 5.2. The need for a ‘macro’ safety case (A) Description of emerging need: In aviation and ATM industry safety assessments have focused on individual concept elements, rather than on the joint effect on safety of multiple changes in air traffic operations. SESAR is defining advanced developments to air traffic operations, consisting of multiple local changes by various stakeholders. As the relations and interactions between such individual operational changes need to be properly assessed, the need for a ‘macro’ safety case is identified in [SESAR SMP]. Such ‘macro’ safety case has a dual character: at one hand interactions between different operational improvements need to be analyzed on safety, at the other suitable safety targets need to be defined for parts of the novel operation. Description of identified approaches: The following complementary approaches have been identified that aim to address this need: 1. Integrated Risk Picture (IRP), as an overall incident-accident model of the ATM system organizing and

integrating safety assessments for individual operational changes;




2. Apportioned ATC safety criteria based on accident rates [Van den Bos et al., 2009]; and 3. Performing ‘joint safety analysis’ using TOPAZ. Re 1: In [Fowler et al., 2009] it is explained how an Integrated Risk Picture (IRP) is used for the SESAR operational concept as overall incident-accident model of the ATM system. This way, safety assessments for individual operational changes are organized and integrated, covering their functional interactions and common causes. Accordingly, this forms a top-down approach considering the ATM system as a whole, complementing a bottom-up approach to assess risks associated to hazards that are either affected or newly generated by the introduction of each individual operational change. IRP is introduced in more detail in [Perrin et al., 2007], where it is explained how a ‘baseline’ (IRP 2005) and a future risk picture (‘predicted’) version of IRP have been developed. The ‘predicted’ version aims to model the safety impacts of all known ATM changes, leading to an indication whether the safety targets can be achieved, and enabling apportionment of an overall safety target based on the overall ATM contribution to aviation accident risks, by assuming the modelled performance of individual ATM elements as safety objectives for safety assessments for individual operational changes. To ensure safety also between the baseline and the eventually foreseen situation, the use of IRP is complemented by a ‘Safety Targets Achievement Roadmap’ [Vernon & Perrin, 2007], which takes into account traffic growth and the foreseen implementation planning. Re 2: In [Van den Bos et al., 2009] apportioned ATC safety criteria are presented that are based on accident rates. The focus is on ATC-related accidents, being all accidents that ATC should prevent. This way, all accidents related to separation provision are considered, irrespective of which stakeholder (e.g., ANSP, airline) has causal contributions to the risk. An overall safety target for ATC-related accidents is apportioned into safety targets on the level of so-called ATC sub-products, which are comparable to parts of a flight forming a logical element within an ATC service or unit (e.g., ‘taxiing’, and ‘line-up’). Individual safety assessments consider one or more operational improvements and connect to the level of the ATC-sub products. Re 3: TOPAZ has been developed for ‘joint safety analysis’ of advanced air traffic operations. It addresses all types of safety issues, including organizational, environmental, human-related and other hazards, and any of their combinations. Notably, it also considers all relevant stakeholders in an integrated way, enabling to cover well interactions such as between pilots an Air Traffic Controllers. It makes use of safety relevant scenarios that model the combinatorially many possible interactions between hazards and elements under control by different stakeholders. It features development and subsequent use of a Monte Carlo simulation tool set for selected parts of advanced operations. For other parts and other design options, possibilities are to adopt a qualitative approach, to use sensitivity analysis of a simulation, to rerun simulations with adapted parameter settings, and to cover it via an advanced bias and uncertainty assessment. Guidance on use of complementary approaches: The key value of the IRP approach is that it aims to cover the wide variety of operational changes foreseen by SESAR. Also, it is fully compatible with a system engineering approach. The apportioned ATC safety criteria may provide a valuable complementary view regarding various flight phases. TOPAZ may deliver complementary value by a better coverage of interactions between nominal and non-nominal behaviours of different elements. Also TOPAZ provides feedback with the focus on understanding the main sources of safety risk, and on sensitivity of risk to changes in parameter values. These sensitivity values allow system design experts to learn how changes in safety requirements affect safety risk. This specifically will deliver added value for advanced concepts where the dynamic interactions between multiple actors plays a key role in the safety assessment, and the changes over the current practice are significant.




5.3. The need to address safety regulations (B) Description of emerging need: Even though ATM safety regulations have contributed to the successful delivery of an acceptably safe ATM system across Europe so far, significant issues exist with respect to the current regulatory framework. Main conclusion of [SESAR WP1.6.1/ D1] is that developing the ATM safety regulatory framework will be essential to the success of SESAR. This improvement should be aimed at providing a clear, unambiguous set of regulations integrated with the safety regulation of the other parts of the air transport industry. Main issues are in the field of: • Fragmentation and variability in regulations and their interpretation. • Safety accountability: The complex safety regulatory framework and the often detailed and prescriptive

nature of safety regulations can result in confusion over safety accountability; • Duplication of regulations: overlap and contradictions leading to confusion and difficulty; • Complexity of regulation, leading to difficulty to comply; • Transparency, as ATM regulations are frequently too detailed and prescriptive in nature; • Harmonisation of industry regulation, with a lack of harmonisation in safety regulation in air transport,

while conflicts in regulatory requirements could lead to safety being compromised; • Proportionality and cost effectiveness: it is not possible to determine whether ATM safety regulation is

cost-effective, nor whether resources are being deployed in a way that will minimise risk. In summary, safety regulations need to be properly addressed. As an indication that this safety regulation may further evolve, SRC has recently started the development of dedicated tools for the provision of safety oversight in the early phases of R&D [SRC CG STF]. Description of identified approaches: The following approaches have been identified that aim to address this need: 1. Methods for early scanning of concepts on Safety Fundamentals; 2. Identification of complementary regulation needs, and showing via safety assessment the impact of

improving standing regulations or not; and 3. Performing safety assessment assuming current regulations, and laying down needs for changes in

assumptions. Re 1: Safety Fundamentals [Safety Fundamentals] reflect a framework of basic safety rules that are independent from the design implementation. The main four aspects of safety considered in this framework are safety regulation, safety management, operational safety and safety performance. An early scanning on safety fundamentals can be used to pro-actively consider safety early in the development lifecycle of an operational concept, potentially leading to amongst others the identification of needed or anticipated changes in safety regulations. Hence, early scanning on safety fundamentals helps in properly addressing safety regulations in concept development and validation. A detailed description of Safety Fundamentals is provided in [CAATS II D13], Part 2, Appendix VI. Specific methods exist to support the analysis of Safety Fundamentals, see e.g., [SESAR 1.6.2/ D3] and [Strater et al., 2007]. Re 2: In the draft [RESET D7.1] it is argued that identified needs for improvement of the safety regulatory framework impact safety assessment in R&D. A concept under study in R&D will eventually need to be proven sufficiently safe according to the safety regulatory framework that will be in force at the time of certification and implementation of the concept. A concept will thus have to show that it satisfies the requirements of a future safety regulatory framework. As this future safety regulatory framework is not yet available, this forms a ‘moving target’ for safety assessment in R&D. It is not straightforward to deal with this moving target for safety assessment in R&D. In the draft [RESET D7.1], it is first identified which changes are needed in the safety regulatory framework to enable successful development of considered




concepts with decreased separation. Next, safety assessment is proposed according to current requirements and requirements from anticipated changes in regulations. Re 3: In [SESAR SMP] it is proposed to perform safety assessment in line with current regulations. For items not yet covered by current regulations it proposes to work with assumptions and safety requirements, which may next be adopted in complementary regulation. Guidance on use of complementary approaches: It is advisable to scan concepts on Safety Fundamentals in an early E-OCVM phase. This way, needed changes in safety regulations may be identified. Depending on the significance of the needed changes in safety regulations, it can be advisable to anticipate on changes in safety regulations in the further safety validation. The alternative is to start the safety validation in line with current regulations, as long as one has verified that this does not block healthy innovations in aviation. 5.4. The need to address the multi-stakeholder natu re of advancing air traffic operations

(C) Description of emerging need: The SESAR operational concept will introduce significant changes to the way in which ATM is performed. The concept will fundamentally change the roles of many of the stakeholders in the ATM system and, importantly, these roles will change dynamically within the operation as a flight progresses. This will result in new ATM safety responsibilities and new interfaces between stakeholders. Examples of such changes are in the field of: • Airspace Organisation & Management; • Separation Provision • Collision Avoidance Necessary precautions should thus be taken to ensure an appropriate approach towards safety for SESAR in its widest sense, to enable an acceptably safe implementation of the SESAR concepts, to minimize SESAR project risks and related costs, and to support the EC and SJU in their respective requirements to provide information and the discharge of their explicit responsibilities and accountability towards safety in ATM. From these conclusions from [SESAR WP1.6.2/ D3] the emerging need is identified to properly address the multi-stakeholder-nature of advancing air traffic operations. Description of identified approach: SAFMAC has been identified as approach that aims to address this need. SAFMAC [SAFMAC] is a safety validation framework, which has been developed to incorporate into safety validation the active roles that have to be played by stakeholders during the development phases of a major change in air transport operations. In its detailed alignment with E-OCVM, the focus during the R&D phases (V0 to V3) is on the macro level of institutional conditions, i.e., the interactions between stakeholders’ organisations and operational control. Key issue is that during R&D the stakeholders should jointly adopt a goal oriented approach. This is put in practice via iteration of four processes, in which joint goals are set (set goals), conops versions are developed to reach these goals (plan), the consequences for the stakeholders are identified (act), and the conops version is jointly validated (joint safety validation). The joint safety validation should make sure that emergent behaviour from interactions between the stakeholders is properly addressed. It is noted that the TOPAZ methodology has been used since its development for such joint safety validation. A detailed description of SAFMAC is provided in Part 2, Appendix VII. Guidance on use of complementary approaches: SAFMAC has been aligned with E-OCVM. Its active involvement of stakeholders in development and validation, and explicit synchronization of stakeholder goals deliver complementary value for advanced multi-stakeholder developments.




5.5. The need to address the success side of a chan ge (D) Description of emerging need: In aviation and ATM industry safety assessments have focused on what happens if a new or changed system fails in some way. The potential positive contribution of the change is often left unaddressed. Similarly, instead of focusing on failures of ATM only, the positive contribution of SESAR to aviation safety should also be considered. Therefore, in [SESAR SMP] the need is identified to address the success side of a change. Description of identified approaches: The following approaches have been identified that aim to address this need: 1. TOPAZ; and 2. SAME. Re 1: Since its development the safety assessment methodology TOPAZ considers success and failure in an integrated way, and hence forms a proven approach to covering both the success and failure side of a change. The method uses ‘safety relevant scenarios’ in which it is modelled how the resolution of hazardous situations depends on the performance of multiple elements, acknowledging that performance variability goes further than the occurrence of failures, and that this plays an important role in safety. Re 2: Safety Assessment Made Easier (SAME) ([SAME PT1], [Fowler et al., 2007]) is developed by EUROCONTROL as an extension of SAM. Where SAM focused on the negative contribution to risk, SAME also considers the positive contribution of the concept under investigation to aviation safety. It does this by proposing a ‘broader approach to safety assessment’, consisting of complementary success and failure approaches: • The success approach seeks to show that an ATM system will be acceptably safe in the absence of

failure; • The failure approach seeks to show that an ATM system will still be acceptably safe, taking into

account the possibility of (infrequent) failure. In SAME the safety assessment is driven by a safety argument structured according to system assurance objectives and activities. In [Fowler et al., 2009] the use of SAME for the SESAR operational concept is explained. A detailed description of SAME is provided in Part 2, Appendix V. Guidance on use of complementary approaches: The advantage of SAME is that it builds further on SAM, which is the most frequently used safety assessment methodology in ATM, and at the same time explicitly widens the scope to include also the positive side of a change. TOPAZ has the advantage of allowing an integrated view of success and failure, covering all types of interactions between nominal and non-nominal behaviours of different elements. The use of TOPAZ would specifically deliver added value for advanced concepts where the dynamic interactions between multiple actors plays a key role in the safety assessment. 5.6. The need to cover performance of human operato rs (E) Description of emerging need: In future concepts proposed by SESAR, ATM will remain to be driven by the role of human operators. Therefore the safety of air traffic operations will remain to depend on the role of human operators. So far, many safety techniques have not comprehensively covered for the role of the human operators in the ATM system [SESAR SMP]. There is thus a need to cover performance of human operators appropriately in safety assessments. Description of identified approaches: The following approaches have been identified that aim to address this need: 1. Eurocontrol’s Human Factor case




2. Human Assurance Levels (HALs) 3. Controller Action Reliability Assessment (CARA) 4. Human performance modelling in TOPAZ 5. Resilience engineering approaches 6. Organisational safety modelling Re 1: Eurocontrol’s Human Factor case [HF case] is a process to systematically manage the identification and treatment of Human Factor issues as early as possible in a project’s lifecycle. In the CAATS II project, this Human Factor case has been formalized [CAATS II, D17] for use in line with E-OCVM in the R&D phases. Re 2: In SAM, the use of Human Assurance Levels (HALs) is explored, which aim to ensure an appropriate level of Human Factors consideration/ integration in the system design and working practices commensurate with the risk for a particular system function relying on human performance. Usually, these HALs are used at the leafs of fault/ event trees. SAME, which is incorporating SAM, also proposes the use of HALs. Re 3: CARA (Controller Action Reliability Assessment, [Gibson & Kirwan, 2008]) is a human reliability assessment technique, which can be used to quantify human reliability aspects as failure rates and success of mitigation actions in the context of Air Traffic Management (ATM). Re 4: TOPAZ uses systemic modelling that includes modelling of human performance (e.g., [Stroeve et al., 2009]). Motivation for this is that covering human actors via probabilities in fault and event tree approaches has the serious limitation that this way the impact of concurrent and dynamic behaviour on risk cannot effectively be taken into account. To incorporate for interactions between multiple human actors, TOPAZ includes modelling of multi-agent situation awareness [Stroeve et al., 2003]. Also, it has been evaluated [Blom et al., 2005] regarding possible integration with the human performance model in Air Midas (Air Man-machine Integration Design and Analysis) [Corker, 2000]. Re 5: Resilience engineering [Hollnagel et al., 2006]. Resilience engineering acknowledges that safety does not only depend on risk related to breakdown or malfunction, but also on the ability of a system to adjust to current conditions, which continuously change due to the complexity of air traffic operations. Both the human cognition contribution to resilience (e.g., via coordinating in unforeseen hazardous situations) and possible technological means (see e.g., [Di Benedetto et al., 2008]) that help the human in detecting and restoring from latent conditions which undermine the resilience effectiveness of human operators (e.g., tools that help the operator detect hazardous situations resulting from differences in situation awareness). Re 6: Modelling of organizations organizational safety modelling for ATM is being studied in [Stroeve et al., 2008]. This goes one step further than modelling humans and interactions between multiple humans, in the sense that groups of humans and interactions within and between groups are also considered. Guidance on use of complementary approaches: The human factors case is already part of E-OCVM, and it intends to make sure that Human Factors issue are timely identified and appropriately addressed in development and validation. These Human Factors issues include issues related to safety and issues unrelated to safety. To assure safety, human performance should however also be appropriately addressed in the safety case. A human factors case typically will provide useful input to safety case development. In safety case development by ANSPs the qualitative approach of using HALs may be sufficient in some cases. However, for more demanding applications, there is a need for more powerful means, like the use




of CARA for human reliability assessment. In advanced developments in which interactions and dependent behaviour of multiple operators plays a role, more detailed human performance modelling as done by TOPAZ may be necessary. Attention should be paid to new concepts not loosing resilience of the current operations, and preferably improving on resilience. Specific resilience engineering techniques or models may be selected on a case by case basis. Finally, when changes to organizational structures play a role, organizational safety modelling may be applied. 5.7. The need to identify unknown ‘emergent’ risks (F) Description of emerging need: With the introduction of advanced developments as aimed for by SESAR, yet unknown ‘emergent’ risk may appear. New behaviour and new hazards will emerge that have not yet been seen in ATM. As hazard identification is a crucial step in safety assessments, the need to identify unknown ‘emergent’ risks is identified from [SESAR SMP]. Description of identified approaches: With the introduction of advanced developments as aimed for by SESAR, yet unknown ‘emergent’ risk may appear. Such risk is related to ‘emergent behaviour’ which is characterized by what the interaction between multiple local behaviours (both nominal and non-nominal) yields more than the sum of the local behaviours. The following approaches have been identified that address identification of emergent risk: 1. HAZOP 2. A “pure” brainstorming approach 3. Real-time simulations 4. Stochastic modelling and Monte Carlo simulations. Re 1: HAZOP (Hazard and Operability study, see for instance [Kletz, 1999]) is used for identifying, analyzing and mitigating hazards in sessions with operational experts. The identification is done via brainstorming along keywords. Whereas the more classically adopted hazard identification approach of functional decomposition is directed towards identification of failures of individual functions, HAZOP also enables identifying emergent hazards. Re 2: The ‘pure brainstorming’ hazard identification approach (originally developed in [De Jong, 2004], but also incorporated in [SAM]), puts large emphasis on identification of hazards that are functionally unimaginable, e.g., because they are associated to systems functioning well (e.g., controllers over-relying on new alerting systems), because they are only remotely associated with failures (e.g., differences in situation awareness, variations in effectiveness of conflict detection and resolution), or because they are related to implicit functions relevant for safety only recognized after failure. In [De Jong et al., 2007] it is explained that the ‘pure brainstorming’ approach can drastically increase the effectiveness of HAZOP by keeping hazard identification separate from hazard analysis and risk mitigation. Re 3: Real-time simulations (e.g., using SAFSIM) may be used for identification of emergent risk, including risk related to the emerging dynamics and interactions of the various elements in foreseen air transport operations. Non-nominal events can often be inserted in the simulations, enabling the identification of further, related, emergent behaviour. Real-time simulations can vary in scale, and regularly serve multiple validation objectives simultaneously. Still, the usually low number of runs plays a role in the efficacy in identification of emergent behaviour. Re 4: Stochastic modelling and Monte Carlo simulations: In this approach, first a stochastic model is developed, which is next subject of large numbers of Monte Carlo simulations. The results of these




simulations allow risky behaviour to come to the surface. Only once these are known, it is analyzed from which interactions between which local behaviours this risky overall behaviour stems. This approach allows the identification of emergent behaviour due to interactions of local functional behaviour and also various non-nominal effects, including local issues as situation awareness, human performance, and random effects (e.g., weather). [De Jong et al., 2007] presents an example of identification of high-risk emerging system behaviour from a background of combinatorially many possible system behaviours with lower risk. Note: Classical approaches to identification of hazards are usually more focused on identification of risk related to individual system elements. Search of literature, reporting systems and databases is usually a good mean to complement such identification. Whereas such searches may not be very effective in identifying emergent risk, search of similar safety studies might be a useful complementary mean for identification of emerging risk. Guidance on use of complementary approaches: The strengths of all four identification approaches are best combined, especially for more demanding applications. Insertion of hazards in real-time simulations can be used for identification of emergent behaviour in the detection and resolution of hazardous situations. HAZOP and the ‘pure brainstorming’ hazard identification approach are both useful for identification of emergent behaviour leading to hazardous situations, and emergent risky behaviour in the resolution of such situations. With the ‘pure brainstorming’ approach, the focus is more on efficacy of the hazard identification activity, without already trying to assess and mitigate the identified hazards. Accordingly, this may be more appropriate for advanced concepts. Stochastic modelling and Monte Carlo simulations allow risky behaviour coming to the surface due to interactions of local functional behaviour and also various non-nominal effects. The advantage here is that large numbers of possible interactions between many types of local system behaviours can be tackled. One may incorporate results from other identification means in such a model, or use those results for validation purposes. 5.8. The need to address E-OCVM requirements (G) Description of emerging need: [SESAR CVM] identifies E-OCVM as a common approach to all projects contributing to the validation of operational concepts, and takes it as the basis of the SESAR concept validation methodology. As safety assessment in R&D is done as part of a general validation process, there is a need to address E-OCVM requirements. In particular, E-OCVM puts requirements on the output of safety case development at the end of the phases V0, V1, V2, and V3 of the validation process. Description of identified approaches: As safety assessment in R&D is done as part of a general validation process in which stakeholders need to be regularly informed, there is a need to address E-OCVM requirements. Only since recently, it has been studied how to tailor safety assessment methods on the basis of the maturity of the concept under investigation, e.g. what should be done for concept at phase V1 of E-OCVM, what at phase V2 and so on. Table 5-2 summarizes which sources present general validation views per phase of E-OCVM, and Table 5-3 summarizes which sources present a view on safety validation per phase of E-OCVM.

Table 5-2: Overview of phases for which sources present a general validation view per E-OCVM phase

source ATM need (V0) Scope (V1) Feasibility (V2) Integration (V3)

SESAR DS X X X SESAR SEM X X X





SESAR CVM X X X

SARD X X X X

Table 5-3: Overview of phases for which the sources present a view on safety validation per phase of

E-OCVM. Phases that are not applicable due to the maturity of the considered concept are marked

with n/a.


SESAR WP 1.6 X

SESAR SMP X X X X

SAME X X X X

SAFMAC X X X X RESET X n/a n/a

iFly X X X In Appendix II of [CAATS II D13], an analysis is presented of those views that were available on the date on which collection of information by CAATS II closed. For each phase, the leading view from E-OCVM is presented, as well as the commonalities with and complementarities to E-OCVM of other sources’ views. From the analysis, it is concluded that knowledge on working with E-OCVM in safety assessment in R&D is only just building up, and that there are several candidate approaches for addressing E-OCVM requirements. Guidance on use of complementary approaches: Detailed guidance on use of E-OCVM is given in Section 3.4 of this document. 5.9. The need to assess concept maturity (H) Description of emerging need: E-OCVM provides a sound common foundation for the lifecycle definition of R&D projects that begin since the very early immature concepts and develop until late stages of implementation. In [SESAR DS] it is put forward that it is essential to assess the level of maturity of the subject of the lifecycle before moving to the next phase in the lifecycle, and that decision points should be established to assess the level of maturity and to decide whether to go through the next phase. Accordingly, there is a need to assess concept maturity. Description of identified approaches: The following approaches have been identified that aim to address this need: 1. Strategic Assessment of ATM R&D results (SARD); 2. Recent draft improvements to SARD by researchers involved in SARD and CAATS II; 3. Transition criteria developed as part of SAME. Re 1: Strategic Assessment of ATM R&D results (SARD) [SARD] defines a process and a set of criteria per phase of the Concept Lifecycle Model for the analysis of ATM R&D results per operational concept from a strategic view point. Detailed information about SARD is contained in Part 2, Appendix IV.2.




Re 2: Recently, researchers responsible for SARD and for CAATS II have jointly developed an improved and updated revision of SARD’s detailed criteria [SARD criteria]. In addition, Safety case specific guidance has been developed for the SARD criteria (see [CAATS II D13], Part 2, Appendix IV.3. Re 3: Transition criteria have been proposed for use with the SAME approach [SAME appendix]. Detailed information about the transition criteria proposed by SAME are reported in [CAATS II D13], Part 2, Appendix V.2. Guidance on use of complementary approaches: The guidance presented in Section 3.4 on using E-OCVM for safety case development highlighted the role that transition criteria can play in assessment of concept maturity. Firstly, transition criteria can facilitate determining the maturity of operational concepts in terms of the phases of E-OCVM’s Concept Lifecycle Model. Secondly, they can assist in evaluating the outputs of validation projects, as they allow to measure whether the validation objectives of a specific phase of the Concept Lifecycle Model have been reached. Projects and programs are recommended to build experience with all the recently developed transition criteria described above, as they provide useful tools for determining the maturity of a concept and for evaluating validation results. [CAATS II D13] provides further information on SARD and on the above developments. As discussed, safety development of major, SESAR-like changes poses several emerging needs. When evaluating the maturity of an advanced concept, it should be analyzed whether these emerging needs have indeed been addressed in safety case development. Ideally, the transition criteria used should facilitate this. The safety case specific guidance developed for the SARD criteria already aims to address part of these emerging needs. The developers of all these sets of transition criteria are recommended to pay special attention to accounting for these emerging needs in future updates of these criteria. 5.10. The need for managing relations between cases (I) Description of emerging need: In [SESAR RLP], the need for an integrated management approach is identified which manages safety and other performance areas as business and environment in an integrated way. In the R&D phases, management of performance is organized via E-OCVM’s case-based approach, in which cases are used to • Provide preliminary feedback helpful to reduce the risks associated with a new concept; and • Structure the evidence into a presentable format that helps stakeholders identify the answers to their

key questions. Cases are usually managed by specialists in the domain investigated, for example, the human factor case is managed by specialists on human factor, safety case by safety analysts and so on. In addition, the different domains have different methods and techniques, usually at different levels of consolidation. The consequence of this partition of the work, together with the different levels of maturity of methods and techniques, can be a complete separation of cases from each other. Accordingly, there is a need to manage relations between cases. Description of identified approaches: Appendix III of [CAATS II D13] describes the state of practice regarding relations between cases in the current R&D projects, with special attention to the relations between human factor and safety case. This includes a CAATS II developed framework for relations between cases that was proposed and refined with the collaboration of several experts of the different cases concerned (safety, human factor, environment, business). This work leaded to the preliminary identification of the phases in which different cases should interact with each other and of the type of information they should exchange. The proposed approach and the preliminary results achieved are




presented in Part 2, Appendix III and in reference [CAATS II, D28]. For the specific relations between safety and human factor cases some results are also presented in [CAATS II, D17]. The RESET project [RESET D7.1] aims to use this relation framework in specific for the relations between the safety and the human factor case in the phase V1 of E-OCVM. Guidance on use of complementary approaches: Based on the CAATS II material on managing relations between cases, it is recommended to define relations between the safety case and the other cases, such that overlap between the cases is prevented, and maximum effectiveness and efficiency of the validation process is promoted. Such relations can be defined at the following levels: • Relations between the key outputs of cases, which together inform decisions taken by stakeholders

(e.g., about transition to the next maturity level); • Relations between the main bodies of evidence supporting these key outputs; • Relations between the processes used to create the evidences and the outputs; and • Relations between the teams who are responsible for developing these cases. The following generic relations can be defined for these levels: • For key outputs:

o Coherence of outputs of cases; o Complementarities of outputs of cases;

• For body of evidence: o Consistency of evidences; o Dependencies between evidences;

• For the processes: o Shared data gathering; o Sharing of data;

• For case teams: o Shared understanding; o Shared inputs, information, and concerns.

The use of relations between cases is little exploited. [CAATS II D13] provides an overview of emerging practices and new proposals regarding relations between cases, as one cannot yet speak of ‘good practices’ in relations between cases. Based on this, the following preliminary guidance is given: • General validation exercises, such as simulations or operational trials, can be used for gathering

information for all cases. • Overlap between the various cases should be prevented, and each other’s strengths should be

exploited. • Regarding relation with the human factor case:

o The safety case developers need human factor expertise. o Ideas exist on co-ordination of the safety case and human factor case process, indicating

where exchange of information is recommended, and where activities are better done separately.

• Regarding relation with the business case: o The business case uses input from the safety case. The cost of the new concept depends on

the identification of unsafe elements in the safety case, as these unsafe elements potentially need costly mitigations, or redevelopment. Furthermore, safety gains or losses caused by the introduction of the new concept must be taken into account in the business case.

o Models for assessing the economic value of the safety gains or losses caused by the introduction of the new concept are emerging.

• Regarding relation with the environment case:




o No input-output relations have been identified. o It should be considered which of these two cases considers third party risk part in its scope.

It is recommended to explore the use of the generic relations between cases above, the preliminary guidance described above, and the individual practices identified in [CAATS II D13], in ATM R&D validation projects. This way, projects can start taking benefit from the promising perspectives of defining relations between cases, while at the same time the experience gained can be used to evaluate, improve, and consolidate the practices in [CAATS II D13] and the preliminary guidance above.




6. Concluding remarks E-OCVM, the standard European framework for validation of operational concepts in ATM R&D, has recently formalized the use of ‘cases’ in ATM R&D. These ‘cases’ have the objective to group information into common aspects in order to describe the potential of concepts under evaluation, and thereby support the key stakeholders as they make the investment and implementation decisions, along the R&D part of the lifecycle of a concept. One of the E-OCVM cases addresses safety; other example cases are for human factors, for business and for environment. Whereas the development of a safety case by an ANSP for a change to its ATM system (including humans, procedures, and technical equipment) has become common practice, safety case development in R&D has been subject of a lot of recent research. An analysis of this research is documented in this guidance document’s companion document [CAATS II D13], Parts 1 and 2. From that analysis it appears that: • experiences with developing a safety case in E-OCVM are just building up; • several needs are emerging for safety case development for advanced developments such as aimed


emerging needs. This document has provided guidance on safety case development in ATM R&D while these new complementary approaches are still in development, and while it is not yet fully clear how they can be integrated in tackling the multiple emerging needs of safety case development of large, SESAR-like changes. The guidance presented consists of guidance for defining safety case activities in line with the E-OCVM framework, guidance for using safety analysis, and guidance for addressing the SESAR-identified emerging needs. The following key recommendations have been made: • Experiences should be gained with emerging methods; and • Integration of the emerging methods should be continued in order to combine their strong points.




7. References

[AP15 ppt] Henk Blom, Advanced Safety Methods for use by SESAR and NEXTGEN from 2012 onwards, presentation provided to FAA/Eurocontrol, Safety Action Plan15, version 10 April 2009.

[AP15 Toolbox] FAA/Eurocontrol, ATM Safety Techniques and Toolbox, Safety Action Plan 15, Issue 2, http://www.eurocontrol.int/eec/gallery/content/public/documents/ EEC_safety_documents/Safety_Techniques_and_Toolbox_2.0.pdf October 3, 2007.

[Blom et al., 2005] Blom HAP, Corker KM, Stroeve SH. Study on the integration of human performance and accident risk assessment models: Air-MIDAS & TOPAZ. Proceedings 6th USA/Europe ATM R&D Seminar, Baltimore, USA, (http://www.atmseminar.org/past-seminars/6th-seminar-baltimore-md-usa-june-2005/papers/paper_098), 2005

[CAATS II D13] Alberto Pasquini, Jelmer Scholte, Mariken Everdij, Henk Blom, Bas van Doorn, CAATS II deliverable D13, Good practices for safety assessment in R&D projects, version 3.5, 8 October 2009.

[CAATS II website] CAATS II website: http://www.caats2.isdefe.es

[CAATS II, D17] CAATS Consortium, “Guidance document for a typical HUMAN FACTORS case”, CAATS II Deliverable D17, May 2009.

[CAATS II, D28] CAATS Consortium, “Guide to a comprehensive incorporation of environmental, cost-benefit,, safety and human factors cases in the validation of ATM R&D projects”, CAATS II Deliverable D28, Jun. 2009.

[CAATS Safety] R.B.H.J. Jansen, Good practices and needs for improvement for safety key elements in Air Traffic Management, Final version, April 2006.

[CAATS, D1.4 P2] M.H.C. Everdij, H.A.P. Blom, Safety assessment methodologies, CAATS Deliverable D1.4 safety report, Part 2, 2006.

[Corker, 2000] Corker, K. (2000), Cognitive Models & Control: Human & System Dynamics in Advanced Airspace Operations, Eds: N. Sarter and R. Amalberti, Cognitive Engineering in the Aviation Domain, Lawrence Earlbaum Associates, New Jersey.

[De Jong et al., 2007]

H.H. de Jong, H.A.P. Blom & S.H. Stroeve, How to identify unimaginable hazards? In: Proc. of the 25th ISSC, Baltimore, Maryland, August 13-17, 2007.

[De Jong, 2004] H.H. de Jong, Guidelines for the identification of hazards; How to make unimaginable hazards imaginable? NLR Contract report 2004-094 for EUROCONTROL, March 2004, included in [EATMP SAM, 2007]: FHA, Ch. 3, GM B.2.

[Di Benedetto et al., 2008]

M.D. Di Benedetto, A. D’Innocenzo, A. Petriccone. Automatic Verification of Temporal Properties of Air Traffic Management Procedures Using Hybrid Systems. 7th EUROCONTROL Innovative Research Workshop & Exhibition. December 2-4, 2008. EUROCONTROL Experimental Centre, Paris , France .

[EC 549/2004] European Commission, Regulation (EC) No 549/2004 of the European Parliament and of the Council of 10 March 2004, Framework for the creation of the Single European Sky (the framework regulation).

[ED-78A] ED-78A/DO264 -“Guidelines for approval of the provision and use of Air Traffic Services supported by data communications” EUROCAE, December 2000. (This




document is identical to the US equivalent RTCA DO-264)

[E-OCVM] Eurocontrol EATMP, European Operational Concept Validation Methodology, version 2.0, 17 March 2007

[ESARR 4] Eurocontrol Safety Regulatory Requirement (ESARR), ESARR 4 Risk Assessment and Mitigation in ATM, Ed. 1.0 April 2001, http://www.eurocontrol.be/src/index.html (SRC deliverables).

[Fowler et al., 2007]

Derek Fowler, Gilles Le Galo, Eric Perrin, Stephen Thomas, So it’s reliable but is it safe? A more balanced approach to ATM safety assessment, available at: www.atmseminar.org/all-seminars/atm-seminar-2007/paper_041.

[Fowler et al., 2009]

D. Fowler, E. Perrin, R. Pierce, A systems-engineering approach to assessing the safety of the SESAR Operational Concept 2020 Foresight, Eighth USA/Europe Air Traffic Management Research and Development Seminar, 2009

[Gibson & Kirwan, 2008]

Gibson, W.H. and Kirwan, B., Application of the CARA HRA Tool to Air Traffic Management Safety Cases, EEC May 2008, http://www.eurocontrol.int/eec/gallery/ content/public/document/eec/conference/paper/2008/002_Application_of_CARA.pdf

[HF case] Eurocontrol EATM The Human Factors Case: Guidance for Human Factors Integration, version 2.0, 29 June 2007

[Hollnagel et al., 2006]

Hollnagel, E., Woods, D. D. and Leveson, N., (Eds.), Resilience Engineering – Concepts and Precepts, Ashgate Publishing, 2006

[ICAO doc 9854] ICAO OCD – Doc 9854

[iFly D10.1i] Henk Blom, iFly Deliverable D10.1i, Initial Validation Strategy/Plan, Version: Draft 0.4

[Kletz, 1999] Kletz T 1999, Hazop and Hazan; identifying and assessing process industry hazards, The Institution of Chemical Engi-neers, 4th ed.

[Perrin et al., 2007] E. Perrin, B. Kirwan, R. Stroup, A systemic model of ATM safety: the Integrated Risk Picture, In: Proc. 7th US / Europe Seminar on ATM R&D, Barcelona, July 2007

[RESET D6.1] Henk Blom, Mariken Everdij, Bas van Doorn, David Bush, and Keith Slater; RESET D6.1: Existing safety assessment methods versus requirements, version 1.0, 7 November 2008.

[RESET D7.1] Henk Blom, RESET WP7.1 Working Document: Managing the E-OCVM phase V1 Preliminary HF and Safety Case building process, version 0.3, 11 March 2009

[Safety Fundamentals]

Oliver Straeter, Managing Safety Proactively – Experiences on the Implementation of the Safety Agenda at Eurocontrol, international conference on Probabilistic Safety Assessment and Management 8 (PSAM 8), New Orleans, Louisiana, USA, 2006.

[Safety Methods Database]

Database containing over 700 safety assessment methods and techniques from various industries, Maintained by NLR, Available at http://www.nlr.nl/documents/flyers/SATdb.pdf.

[SAFMAC] M.H.C. Everdij, H.A.P. Blom, J.J. Scholte, J.W. Nollet, M.A. Kraan, Developing a framework for safety validation of multi-stakeholder changes in air transport operations, Safety Science, Volume 47 (2009), pages 405-420. doi:10.1016/j.ssi.2008.07.021.

[SAM] Eurocontrol, Air Navigation System Safety Assessment Methodology (SAM),




SAF.ET1.ST03.1000-MAN-01, Edition 2.1, 2007

[SAME appendix] Eurocontrol, "Broader approach to safety assessment", contribution received from Nicolas Fota for inclusion as a single appendix into the Appendix part of D14, received 5 May 2009.

[SAME PT1] Eurocontrol, Safety Assessment Made Easier, Part 1 - Safety Principles and an introduction to Safety Assessment Ed. 0.92, 11 July 08

[SARD criteria] Eurocontrol & CAATS II, “SARD Life Cycle Phase Transition Criteria - Annex A v1.8 clean.doc”, distributed by Mete Çeliktin on 9 April 2009.

[SARD] Eurocontrol, Strategic assessment of ATM R&D results, Assessment process & criteria, contact: Mete Çeliktin, [email protected] , Version 1.0.

[SCDM] Eurocontrol, Safety Case Development Manual, DAP/SSH/091, Edition 2.2, November 2006.

[SESAR CVM] SESAR Definition Phase, Concept Validation Methodology, WP4.2/Task 4.2.1, Part of DLT-0710-421-01-00, version 1.0

[SESAR D2] SESAR Definition Phase, Deliverable 2, Air Transport Framework The Performance Target, DLM-0607-001-02-00a, December 2006

[SESAR D6] SESAR Definition Phase, Deliverable 6, Work Programme for 2008-2013, DLM-0710-002-02-00, 2007.

[SESAR DS] SESAR Definition Phase, Development Strategy, WP4.2.1 System Engineering Development & Validation Process/D6, Part of DLT-0710-421-01-00

[SESAR RLP] SESAR Definition Phase, T3.4.6/D5 Regulatory - Legislative Planning, DLT-0710-346-00-05

[SESAR SEM] SESAR Definition Phase, WP4.2.1/D6, System Engineering Methodology, DLT-0xxx-241-0x-0x, Status: DRAFT#02

[SESAR SMP] SESAR Definition Phase, SESAR Safety Management Plan (SMP), WP4.2/Task 4.2.1, Part of DLT-0710-421-01-00

[SESAR WP1.6.1/ D1]

SESAR Definition Phase, Study of safety regulatory framework, WP1.6.1/D1, DLT-0507-161-00-03.

[SESAR WP1.6.2/ D3]

SESAR Definition Phase, ATM safety regulation, SESAR Safety Screening & SESAR Concept, Institutions and Regulations, WP1.6.2/D3, DLT_0000_162_00_07

[SRC CG STF] Eurocontrol Safety Regulation Commission (SRC) CG Scan Task Force – progress report, SRCCG13.02 13/05/2009 item 5.1, working paper, 2009

[SRC DRAHG] Eurocontrol Safety Regulation Commission (SRC), Double Regulation Ad-Hoc Group (DRAHG), Report on the resolution of double ATM safety regulation in Single European Sky states, Ed. 1.0, 23 November 2007.

[Strater et al., 2007]

O. Strater, M. Everdij, J. Smeltink, J. Nollet, J. Kovarova, H. Korteweg, A.. Burrage, "Safety Screening – Experiences in applying a proactive approach to concept development within SESAR, Procs of Eurocontrol Safety R&D Seminar, Rome, Italy, 24-26 Oct. 2007.

[Stroeve et al., Stroeve SH, Blom HAP, Van der Park MNJ. Multi-agent situation awareness error




2003] evolution in accident risk modelling. Proceedings of the 5th USA/Europe ATM R&D Seminar, Budapest, Hungary, (http://www.atmseminar.org/past-seminars/5th-seminar-budapest-hungary-june-2003/papers/paper_067 ), 2003

[Stroeve et al., 2008]

Stroeve, S. H., Sharpanskykh, A., van Lambalgen, R. M. Kirwan, B. Safety culture analysis by agent-based organizational modelling. In Proceedings of the 7th EUROCONTROL Innovative Research Workshop & Exhibition, 2008.

[Stroeve et al., 2009]

Stroeve, S.H., H.A.P. Blom, G.J. (Bert) Bakker, Systemic accident risk assessment in air traffic by Monte Carlo simulation, Safety Science, Vol. 47 (2009), pp. 238-249 (http://dx.doi.org/10.1016/j.ssci.2008.04.003)

[TOPAZ] H.A.P. Blom, S.H. Stroeve, H.H. de Jong, Safety Risk Assessment by Monte Carlo Simulation of Complex Safety Critical Operations, Eds: F. Redmill & F. Anderson, Proc. 14th Safety critical Systems Symposium, Bristol, UK, February 2006, Springer

[Tversky & Kahneman, 1974]

A. Tversky, & D. Kahneman, Judgment under Uncertainty: Heuristics and Biases. Science(185), 1974

[Van den Bos et al., 2009]

J.C. (Hans) van den Bos, Hans H. de Jong, and Roy B.H.J. Jansen, Apportioned ATC Safety Criteria Based on Accident Rates, In: ATC Quarterly, forthcoming, 2009

[VDR] Validation Data Repository available at www.eurocontrol.int/vdr

[Vernon & Perrin, 2007]

G. Vernon & E. Perrin, Methodology report for a Safety Target Achievement Roadmap (STAR), Eurocontrol report, May 2007

Documents

CAATS II C APPROACH TO AIR TRAFFIC S II - TRIMIS · This document provides guidance for safety case development in ATM R&D projects, with a focus on application to advanced operational