Independent Safety Assessment (ISA) Technical Forum

Independent Safety Assessment (ISA) Technical Forum Claire Owens, Manager Safety and Risk Assurance

| 1

An overarching strategy supported by a suite of plans to achieve a 40 year vision for our transport system

| 2

Problem solvingthough co-design and collaboration

| 3

Six outcomes

| 4

How we‘ll measure success

| 5

Objective

To facilitate a collaborative learning environment where knowledge can be shared to improve the effectiveness of the ISA practices at TfNSW and in industry

| 6

Assuring TfNSW across all modes

Transport for NSW

Simon Freeman 2011 Transport for NSW

| 7

Simon Freeman 2011

Assurance

Assurance is a set of structured and planned activities conducted through the asset life cycle providing progressive justified confidence that objectives are being achieved and that the asset is or will be fit for purpose

| 8

/ /

I I

I I

I I

I I I I J I I I

' I ' I I I I I \

\

' \ \

I I

' \

I I

' '

I I

' '

/

,,,/'//

/

---------------Asset Assurance

Engineering

', .........................

RAM

Fitness for purpose hole of life cost

,-/ AEO Model Delivers " ,' --- - - - ----

I /

/

' '

I I

\

' \

/ I

I

\

I

\ \ I

I I

I

I I I I I I

' I I I I I I ,

I I I

I I

The elements of assurance

| 9

An assured transport network Through whole of life the transport assets / system must be assured as:

• Reliable • Safe • Operable • Maintainable • Sustainable • Optimised whole of life cost • Fit for purpose

| 10

Assurance and the asset life cycle

G~tcs m~nagod through CCBs Conf19uratoon Control Boards (CCBs)

Transport Network Assurance Committee (TNAC)

| 11

ASA established the Authorised Engineering Organisation (AEO) Model

| 12

Asset Management

Project Management

~ Safety

Assurance

Environment & Sustainability

Industry Engagement

Systems Engineering

Competency & Capability

Human Factors

Quality & Risk

• ASA Technical disciplines

| 13

Track Civil Electrical En gineering Engineering Engineering

(<~>) x I ••••• 0•0 •• •• ~i(•lfJ -F leet Stations & Telecommunications Signals &

Engineering Buildings Engineering Control Systems

ASA Engineering disciplines

| 14

| 15

Assurance layers Project and TfNSW audits,

TfNSW 3rd Level A ssurance reviews, due diligence a nd acceptance

Other AEO / Organisations

Auditing, i ndependent reviews, 2nd Level Assurance independent validation, ISA

Delivery AEO

1st Level Assurance Engineering process, a ssurance process, risk management etc.

Independent safety assessor (ISA) role in the assurance framework

• Safety assurance regime • Risks associated with changes mitigated SFAIRP

Independentprofessional opinion

• ISA reports to CCB/ TNAC at key life cycle phases Through lifecycle approach

• safety change assessment (ISCA) determines theneed for an ISA

ISA mandated for safety significant

change

| 16

Question 1

In your personal opinion, is the role of the ISA clearly understood by TfNSW and its delivery partners?

1. Yes, TfNSW and its delivery partners understand the role of the ISA

2. TfNSW only 3. Delivery partners have a better understanding than TfNSW 4. No, neither understand the role 5. I’m not sure

| 17

Question 2 Do you agree that the ISA role adds value to TfNSW?

1. Yes 2. No 3. Not sure

| 18

Question 3 Do you agree that the ISA role adds value to industry?


| 19

© Network Rail Consulting

ISA – Looking through the lens ASA Technical Forum – July 2018 Steve Ivey – Network Rail Consulting Richard Adams – Abbott Risk Consulting

2

Introductions

Steve Ivey, Director Safety and Assurance NRC e: [email protected] m: 0467 792 721

Richard Adams, Principal Consultant ARC e: [email protected] m: 0405 377 535

© Network Rail Consulting ASA Technical Forum ISA

mailto:[email protected]

mailto:[email protected]

3

Overview 1. The driver for ISA and why ISA? 2. What is an ISA, its intent and its benefits? 3. ISA AEOs 4. View from the project side 5. Independence with Collaboration 6. ISA approach - Good Practice 7. Risk-based 8. Involve ISA Early 9. Key Learnings

© Network Rail Consulting

The contents of this presentation remains the intellectual property of Network Rail Consulting and may be used only in connection with the brief for which it was submitted. It is specifically forbidden to communicate the contents to any third party without prior permission in writing from Network Rail Consulting, and all reasonable precautions must be taken to avoid this occurring.

© Network Rail Consulting ASA Technical Forum – July 2018

4

The driver for ISA ISA concept in NSW rail industry originates in 2013 reform:

Introduction of the AEO Framework

Establishment of the Asset Standards Authority

Who appoints the ISA?

ISA was intended to: Provide TfNSW with a means of assessing assurance provided by AEOs

Support the asset assurance framework

Aid the acceptance of assets

Assessment of validity of safety argument through lifecycle to provide TfNSW with progressive assurance


5

Why ISA? Represents international good practice

Required by EN50129 (Signaling and Comms)

Mandated under European Common Safety Method for higher risk changes

Included under IESM Guidance

Used in Defense and Nuclear industries around the world

Fitted well with the AEO Framework and post-reform Assurance Model

Required under ONRSR Major Projects Guideline V1.1 July 2016


6

What is an ISA? Independent Safety Assessment is (as defined by the IET):

"………………the formation of a judgement, separate and independent from any system design, development or operational personnel, that the safety

requirements for the system are appropriate and adequate for the planned application and that the system satisfies those safety requirements.”

Independent Safety Assessment performs a key role in the TfNSW assurance framework for transport assets. TfNSW sees ISA as:

“………………the formation of an independent professional opinion of the validity of a safety argument supporting a new or altered asset”


7

The intent of ISA To provide an independent professional judgement of the validity of the safety assurance and safety argument

Support the acceptance of assets under the Configuration Management Process

Support progressive assurance

Provide an additional assurance level for higher risk changes

Add confidence to the assurance process


8

Benefits of ISA Aligns with worldwide good practice in the delivery of assured systems important to safety

Drives high-quality assurance and integration of safety into Transport assets and systems

Provides TfNSW progressive justified confidence in the safety of its assets

Provides additional assurance to ORNSR – the ORNSR Major Projects Guideline requires ISA on major projects

Not just a review of Safety Assurance Documents

Early engagement can drive out potential design Safety assurance issues early


9

ISA AEOs Originated at the time the AEO Framework was being developed

TfNSW needs to have confidence in ISA organisations

Needs to be a level of consistency of approach to support TNAC and CCBs

Allows ongoing auditing of ISAs to drive a maintained level of quality

Support the development of maturity and competence across the industry


10

View from the project side The establishment of the ISA concept is vindicated

Evolving concept from TfNSW and industry’s point of view

Variation in how it is used from project to project

TfNSW very much needs it to drive good assurance outcomes from industry

ISA is definitely a team activity

The industry has accepted it to a degree

It would be beneficial to have a common national approach

May be beneficial to have a national certification body as per Europe


11

Independence with Collaboration Do we think this is acceptable or even possible? ABSOLUTELY

The ASA ISA Guide defines independence as

“………………the assessment body may not become involved as direct or indirect representatives in the design, manufacture, construction, marketing,

operation or maintenance of the system under consideration".


Whilst also stating:

”………………the ISA should adopt a proactive stance in raising questions, requesting additional information or analysis rather than remaining a passive

reviewer of deliverables. A proactive approach helps to identify issues earlier, ensure the context and understanding are developed to ensure the implications

of issues are fully understood and actions can be developed that will address issues comprehensively.”

12

ISA Approach - Good Practice It is Collaborative

Located in the Clients Office if possible

Access to Project Documentation

Regular ISA Forums with Project Teams and Contractors

Joint Scheduling of Project Due Diligence Review and ISA

Liaise with CCB Chairman

Monthly Reporting

Preparation of Safety Notices, ISA Statements and ISA Reports

Independent and/or Collaborative Auditing

Risk-based assessment


13

Risk-based It is not ISA role to review all project outputs but to overlay appropriate level of assurance activities on processes being employed by the Project. It should also include SME review of design and its integration but needs to be risk-based

Risk Assessment is to be determined with focus on areas of highest risk, novelty and complexity

Is continually reviewed as knowledge increases from the assessments undertaken throughout the project lifecycle

Ensures that the ISA activities are not over-burdensome on the project and represent value for money while achieving a level of assurance commensurate with the scale and complexity of the project

Claim Structure (GSN) used to represent the key claims that the assessment activities will focus on


14

Assessment Areas Important to tailor Assessment to the Project but should consider:

Integration of sub packages into an integrated system

Safety Management System and processes

Integration of safety into design and engineering

Management of interfaces – contractors and system elements

Safety risk management

Demonstration that safety is ensured SFAIRP

Management and due diligence of supply chain

Assurance delivered by supply chain including their contribution to all of the above


15

Involve ISA Early 1. It is hi ghly important to engage ISA as e arly in the D esign

Lifecycle as possible.

2. ISA Experience During Concept Phase. Concept Phase is focussed on strategic decision making:

It is only a Reference Design – So safety doesn’t matter. WRONG

How can we possibly say its SFAIRP yet? Design ’towards SFAIRP’

Options Analysis – What about consideration of safety, not just cost?

Who will be responsible for the decisions made

No mechanism for transferring ISA issues to next phase or ISA

Walls between Contractors and ISA’s

Future reliance on Assurance


16

Key Learnings from conducting ISA The Industry does not yet fully understand the role of ISA

Or

The tension between contract and commercial outcomes and suitable and sufficient assurance

System-level versus CCB level in large projects – ‘salami slicing’ of assurance

Competence is a key concern in the provision of assurance

Misunderstanding of the AEO framework


17

Industry doesn’t yet fully understand the role of ISA Commonly appears as part of the safety argument

Often seen as part of the AEO’s quality assurance process

AEO’s don’t always recognise interacting with the ISA is part of their responsibility as an AEO

Do not fully understand where ISA fits in the overall assurance framework

Industry can become frustrated with the ISA and often Seen as ‘Non-Value ’ – does industry see the value of ISA?


18

Tension between contract and commercial outcomes and suitable and sufficient assurance

Contracts continue to be very prescriptive

Key drivers are delivery to time and budget

Industry wants to minimise its cost

ISA isn’t given sufficient time for its assessments – squeezed between AEO and CCB

Assurance is where cutbacks can occur – tends to separate safety activities from the engineering activities

TfNSW still needs to use its own safety assurance teams to drive suitable and sufficient assurance


19

System-level versus CCB level in large projects – ‘salami slicing’ of assurance

Intent of ISA is system level assurance of assets in the transport network environment

Configuration management process is key to TfNSW gaining progressive confidence in the delivery of its assets

CCBs break delivery down into small parts – drives TfNSW and AEOs to expect specific ISA input on small packages of work

Reduced timescales for review required

Risk-based ISA is necessary

ISA needs to be strong and professional

The relationship between CCB / TNAC and ISA is important


20

Competence is a key concern in the provision of assurance

Experience to date – competence issues are very common

AEO versus ISA – industry often questions why ISA looks at competence when it is part of the AEO assessment and audit regime

Consistency required in the management of competency


21

Misunderstanding of the AEO framework

Constant issues with the use of non-AEOs

Must work under the procuring AEOs process

Integrating AEO

Most work is done by consortia or joint ventures

Lack of guidance from ASA on how AEO works in these scenarios

The absence of appreciation of how the whole of life-cycle safety assurance programs should be managed

Integration of the safety argument between TfNSW and its AEOs


Thank you

www.networkrailconsulting.com

ISA assessment and audit updates and changes to ISA documents Claire Owens, Manager Safety and Risk Assurance

| 25

ISA assessment and audit • Assessments 8 organisations – 6 Authorised, 2 in final stages of assessment

• Surveillance audits

3 organisations

| 43

TfNSW projects engaging ISAs

• NIF – New Intercity Fleet • SGT – Sydney Growth Trains • Sydney Metro Northwest • Sydney Metro City and

Southwest • ATP (Automatic Train

Protection) • Advanced Train Control

System (ATCS)

• TTU (Tangara Technology Upgrade)

• Parramatta Light Rail • Sydney Light Rail • Newcastle Light Rail

| 44

Assessment findings AOCs and OFIs • Competence management and maintenance of capability • Definition of roles and responsibilities for ISA • Governance and independence • Deployed evidence as an ISA • Reliance on individuals in an organisation for ISA • ISA only organisations struggling to demonstrate the standard

AEO requirements especially for system safety engineering

| 45

Audit key findings AOCs and OFIs • Escalation and reporting • Communication channels • Reporting • Risk based approach to planning • Traceability of observations to facilitate close out • ISA providing solutions

| 46

Challenges to become an AEO ISA

• Systems and deployed evidence • Deployed evidence for ISA services as an organisation as

opposed to individuals • Limited pool of subcontractors/ resources for the ISA team • A robust competency management system • Takes time for ISA organisations to be assessed • ISA only organisations difficulty to demonstrate system safety

assurance

| 47

Challenges faced engaging an AEO ISA

• Limited number of AEO ISA’s • AEO ISA’s pick and choosing work that has longer duration and

more $’s • ISA remit not issued in the earliest phase of the project lifecycle • ISA remit is not appropriate or relevant for the works • Not always clear when to engage an AEO ISA and who is to

engage the ISA TfNSW or the integrating AEO

| 48

ISA document updates

q"'I NSW Transport - forNSW

T MU MO OOOCU TI

T.c-hnk: ... Information

Independent Safety Assessor (ISA) Requirements (Interim)

VCfSICl'l 10 -°"" 15Mov201< Eftec:llVeDele 15May20H

____ .,, __ ,, ____ ...,... .. __ ..... ____ ..... _ :::=--=.·.-::;-.::: .. -:::.:i..-:.=:;..-----·-.... -· . ..., .... -.... --..... ---···-·--· .......... --.... -. ......... _,, .. __ _ ,_ ___ .. _ c--__________ ..._ __

-··-==· ............. ,_ .....

• 1 NSW Transport -~ forNSW

T MU MD 00003 CU

Guide

Guide to Independent Safety Assessment

... _____ .. _____ _..._. ........ ___ _. ....... ==--......::: .. -:::'".!.'.:t~::i.-.:.=:::...----... ··--··-.--::.-:=-..::=.::...~.:.:··-·--·--------.. -· --;.;---0.-0------.. -............... -.

| 49

-

ISA role and the lifecycle

Planning and business needs

definition

Operational concepts and Metrics: Options, Analysis and

Trade off feasibility study

Concept, functional Architecture

System level design and physical architecture

Subsystem level design

Unit level design

Procurement, fabrication/ manufacturing / construction/

installation

Subsystem level integration & tests

Unit level tests

System level integration and tests

Operational system acceptance tests

Operations and Maintenance changes/ upgrades

De commission and disposal

ISA

AEO Engagement

| 50

ISA role in the lifecycle Early lifecycle

• Key safety decisions • Requirements definition

Design • Progressive assurance • Assurance influencing design • Address any issues or deficiency early to support SFAIRP • Minimise risk is non-acceptance later in lifecycle

Implemention of design V&V – aligned with level of risk

• Not just functional requirements met but integrity requirements are also achieved Key role in acceptance process

| 51

ISAs and Competence Management Richard Shorten, Engineering Competency Development Manager

| 38

Setting the scene

| 53

Key documents

• N~ I Transport -~ forNSW

TMU MOOOCOl.TI

Independent Safety Assessor (ISA) Requirements (Interim)

-·· lllUodO.. t$M9!1i20l-4 E'*M 0. 15 M1Y 2014

:=.=.::::::=.:-=:::.~---=:.:.-:.-:.-:-:..-::-.:::-::.:=--

::.-:--..:::"::..-:-..:::··--··------· ~------------·

f ~ I Transpart - forNSW

T MU MO 00003 GU

Guide to Independent Safety Assessment

::=::.::::::.:::-:=:..i::.:..~:.-:.:.~:..-::::.:-.::..-

:.-::-.:::.::..-:::··--··----·--· ~------.. ----·

e 1 Transport - fo<NSW

T MU CV 10503 OU

AEO Guide to Engineering Competence Management

VertlOl'l 10

Mu9d~ , , Augi.nl1011

| 54

What does competence look like?

Knowledge Experience Attitudes Behaviours

| 55

Rules of evidence

Validity Currency

Authenticity Sufficiency

| 56

| 57

Continuous improvement of the system

Summary

• Be systematic

• Determine competence requirements

• Rules of Evidence

• Establish competence of individuals including associates

| 58

Further information Richard Shorten

A/Engineering Competency Development Manager (02) 9422 7021

[email protected]

| 59

Question 4 Is a national common approach to ISA needed?


| 60

Question 5 Does Australia need a national certification body for ISA like Europe?

1. Yes 2. No 3. Not Sure

| 61

Picture caption

Q & A session

| 48