S.F. Bay Area Rapid Transit 1 Vehicle ATC Safety Certification MMF Monitor Mode Field Release August 28, 2001

S.F. Bay Area Rapid TransitS.F. Bay Area Rapid Transit1

Vehicle ATCVehicle ATCSafety CertificationSafety Certification

MMFMonitor Mode Field Release

August 28, 2001


Presentation Objective

To demonstrate that the required analysis and testing has been completed to ensure safe operation of the VATC system with the design modified for Monitor Mode Field (MMF) operation

To guide the CPUC to through the suite of Certification Documents provided as evidence of safety compliance


Agenda Laying the Groundwork

Safety Assurance Concepts(How can we claim it’s safe?)– For the Baseline Design– For the MMF Release

Verification and Validation Process/ Documentation (What did we do to prove it’s safe?)

Summary





Summary


Laying the Groundwork

-What is the VATC?-VATC Modifications for the AATC System-What are the MMF and CMF Releases?-Overview of the Documentation Package


What is the VATC?Central

StationTrain Control

Vehicle ATC

Other Vehicle Subsystems

Trackside Train Control

OnboardEquipment


What is the VATC?

Computer

Inputs

Outputs

VATC

Station Train Control

Vehicle Sensors Other VehicleSubsystems

Station TrainControl


What is the VATC?

SafetyCritical

Functions

Non-SafetyCritical

Functions

Implemented with design techniques that mitigate hazardousoperation

Implementedwith specialcare


What is the VATC?Vital Door

Control

Vital Braking Control

Non-Vital Motion Control

Non-VitalCommunications

Track Circuits

TracksideCoverboards

OnboardSensors

Trackside Coverboards

OnboardPropulsion/ Brake Controller

Vehicle DoorRelays

Vital control alwaysOver-rules non-vital

VATC





VATC Is A Legacy System

Therfore for the modified system:– Requirement to achieve a MTBH of 250,000

years per unit to apply

– Assumptions of original design still hold

– All functional requirements of the baseline system still in place


VATC Modifications for the AATC System

Existing WaysideTrain Control

VehicleDoor Relays


BaselineFunctions

ExistingOnboardSensors

AATCRadio/ATIC

NewFunctions

AATC controlsOver-ride baselinecontrols

AATC SpeedCommands

VehicleStatus


VATC Modifications for the AATC System


VehicleDoor Relays


BaselineFunctions


AATCRadio/ATIC

NewFunctions


AATC SpeedCommands

VehicleStatusNO HARDWARE CHANGES


VATC S/W Modifications for the AATC SystemExisting Wayside

Train Control

AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus





The MMF Release


AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus

Safety Certified

Implemented but not certified


The CMF Release


AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions

AATC controlsover-ride baselinecontrols

AATC SpeedCommands

VehicleStatus

Safety Certified

Implemented but not certified





Documentation Package Overview

Notice of Intent to OperateMemo from TSD to BART SafetyLetter from Harmon to BART

Verification of Test Readiness

VATC Equipment Configuration SheetVATC S/W, Module Software Configuration

Safety Certification Plan Document List

LETTERS

CERTIFICATE

CONFIG.CONTROL

SCP REQM’TS


4. Safety-Related Tests Verification

5. Hazard Identification and Resolution Verification

Attachments

1. Design Criteria Conformance

2. Specification Conformance Verification

3. Personnel Training Conformance Verification

4. Safety-Related Tests Verification

5. Hazard Identification and Resolution Verification

CERTIFIABLE ELEMENTS


Elements 4 and 5

Element 4 VATC Subsystem V&V PlanVATC Modifications V&V Report

Element 5QA Plan and ReportSafety Assurance ConceptFault Tree AnalysisSafety Assurance Concept Implementation

AnalysisFMEA’sAudit Reports





Summary


Safety Assurance Concepts

-What do we mean by a Safety Assurance Concept?-SAC’s for the baseline system-SAC’s for the modified system-Comparing the two





A Basic Truism For All

Man-Made Systems

ThingsFail

HumansErr

MAN-MADE SYSTEM

CORRUPT OPERATIONS


What Do We Mean By a Safety Case?

ThingsFail

HumansErr

Safety Case

HUMAN LIFE WILL NOT BE ENDANGERED


SystemRequire-

ments

SubSystem

Require-ments

SoftwareModuleRequire-

ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN TRANSLATION

IMPLEMENTATION

EPROMBURNER

Intel8086

ELEMENTS ofDESIGN

OPERATION


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR

RUN TIMEERROR

EPROMBURNER

Intel8086

OPPORTUNITIES forCORRUPTION

IMPLEMENTATION ERROR


Safety Assurance Concepts Required For the VATC Design Design Error

– How do we provide assurances that errors are not made during the creative design phase?

Translation Error– How do we ensure errors are not made during the

systematic translation process? Implementation Error

– How do we protect against errors made while transferring the implementation to the firmware?

Run Time Error– How do we assure that hardware failures and errors in the

non-vital code do not lead to hazardous operation?





SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086

Safety Assurance Concepts- Baseline System-


SAC For S/W Design Errors- Baseline - Highly modularized software segregating safety critical functions from

non-safety critical functions Progressively structured software development process with

disciplined verification of each step of the process

– Westinghouse claimed that all prudent and practical steps had been taken

Westinghouse claim: all prudent and practical steps taken to reduce risk of software errors causing hazard to an acceptable level


Modular Software Development

Module1

Module2

Module3

Module4

RequirementsBook


Modular Software Development

Module1

Module2

Module3

Module4

RequirementsBook

Non-vital

Vital


Execution Flow of Modular Software

Module1

Module2

Module3

Module4


Monolithic Software Development

RequirementsBook

ONE LARGESOFTWAREPROGRAM


Execution Flowof Monolithic Software

ONE LARGE SOFTWARE PROGRAM

GO TO

GO TO

GOTO


SMRS Pseudo Code

RequirementsBook

Progressively Structured Design Validation

PDR Document

PreliminaryDesign Review

Software Design Review

Design WalkThroughs


SAC for S/W Design Errors-Westinghouse Claim-By modular design and a progressively

structured review process, all prudent and practical steps have been taken to eliminate software errors

THEREFOREProbability of the existence of an unsafe software design error is assumed to be zero


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



SAC for S/W Translation Errors

Use of Assembly Language to reduce the risk of COTS Development Tool bugs

Coding Standards developed and used during translation from Pseudo Code to Executable Code

Unit testing on all vital software modules


Assembly Language vs. Higher Order Language

1011100011101101110011001110000110101000

MOV AX, R2

CMP AX, 16

JPE R3

IF X > Y THEN GO TO LABEL

High OrderLanguage

AssemblyLanguage

MachineLanguage

Compiler Assembler


Coding Standards

Guidelines for writing software modules Ensures uniformity across software modules Avoids common coding pitfalls

– Applies to:» SMRS» Pseudo Code» Source Code

;-----------------------------------------------; Check if clock time MSB is ready to roll over.; (Greater than or equal to 0127h);-----------------------------------------------if_2: cmp CX, L622A_MAX_CLOCK_TIME_MSB jae else_2then_2:;-----------------------------------------------; Clock time MSB is not ready to roll over.; If LSB just rolled over, simply increment; the clock time MSB.;-----------------------------------------------if_3: cmp BX, 0 jne endif_3then_3: inc CXendif_3: jmp short endif_2else_2:

Separate standards for each


Unit Testing Performed on individual

vital software modules that are either new or modified

Test results verified that Min/Max/Zero input values produced expected results

Test results verified that all branches in the module were executed

Software Module

All possible Inputs

All Outputs are CorrectAll Branches Checked


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



Implementation

Address Data

00010000

000200030004000500060007000800090010001100120013001400150016

XXXX

0010 11110110 11000101 1100

0001 1010

0111 0000

0010 11110110 11000101 11000001 1010

0111 0000

0000 0000

0101 01011111 0000

0101 01011111 1111

1111 00000001 1101

0011 1111

HexCode

EPROM


SAC for Implementation Errors Double Storage of Vital Program Code

Checksum stored with program code


Double Storage of Vital Program Code

If glitch occurs while “burning” one locations, it is unlikely that the same glitch willoccur while burning the same information in another location

Vital Program

Non-Vital Program

Copy 1

Copy 2

One Copy

EPROM

Faulty EPROM hardware AND/OR faulty software tool will probably not be faultyin the same way at two different memory locations.


Checksum on Program Code

Address Data

00010000

000200030004000500060007000800090010001100120013001400150016

XXXX

0010 11110110 11000101 1100

0001 1010

0111 0000

0010 11110110 11000101 11000001 1010

0111 0000

0000 0000

0101 01011111 0000

0101 01011111 1111

1111 00000001 1101

0011 1111

ADD

CHECKSUM

Then during programoperation, sum contentsof memory and comparewith checksum.

Any mismatch interpretedas program code fault.

EPROM


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



Run Time Errors

PROCESSING HARDWARE

Computer System

InputHardware

sensors antennas receivers

etc.

OutputHardware

driversamplifiers

transmittersEtc.

Vital Software

Non-Vital Software

Software design must protect against hazardous operation in the presence of:

Hardware failures

Corrupt processor behavior caused by errors in the non-vital software


PROTECTION FROM HARDWARE FAILURES-Electrical/Electronic Systems-

System Function

Input Output


Safety Critical Systems

Must not fail and output unsafe result.

Example: Go when you should stop.

OK to say stop when you should go

Contract Requirements = 1 in 250,000 years.

Safety CriticalFunction

Input Output


Before Computers

Input OutputSafety CriticalFunctions

(10’s of components)

- Functions were simple

- Circuit responses to component failures could be analyzed to verify the absence of unsafe failure modes

FAILSAFE


NOW WITH COMPUTERS

Input OutputSafety CriticalFunction

(Millions of components)

- More complex functions

- Impossible to analyze as before

- Other techniques needed:-Numerical Assurance-Checked Redundancy-Functional Self Tests

Applicable to VATC


Checked Redundancy SAC to Protect Against Run-Time Errors

Redundant implementation of critical functions.

Discrepant results not allowed to reach output

System shut down if failure persists

Probabilities calculated to determine MTBH

Input OutputRedundant

Implementation

PeriodicallyChecked

Redundant Implementation

Compare



Implementation

PeriodicallyChecked


Compare

Channel IndependenceA and B must have no common mode failure mechanisms

Channel IntegrityA and B must be correctly implemented

Channel InspectionA and B must be tested periodically

The Three I’s of a Successful Checked Redundant Design



Implementation

PeriodicallyChecked


Compare

Mean Time Between Hazard

Probability of unsafe failure (Pus) occurring during mission time (for VATC = 1 year)

Then MTBH is the reciprocal of Pus

Pus is very much a function of hardware failure rates, the check interval, and the effectiveness of the check



Implementation

PeriodicallyChecked


Compare




Pus is very much a function of failure rates,the check interval and the effectiveness of the check

Software V&V


Test Intervals

Initialization Tests – Performed upon power up

Operational Tests – Performed during operation

Safety Certification Tests – Performed during Periodic Maintenance

TIMELINE

Safety Certification – every 365 days

Initialization - every 60 hrs

Operational–10 s to 1.2 hrs


Test Methodology


Self Testing of all non-failsafe hardware components


Self Testing of non-failsafe hardware components (also referred to as Interleaved Tests)

Cross Comparison of results from redundant channels


Testing with special test equipment


SerialI/O

RAM

WatchdogTimer *

ParallelI/O

ROMInterrupt

Controllers

ParallelI/O

CPU

SerialI/O

RAMROMInterrupt

Controllers

ParallelI/O

CPU

WatchdogTimer *

ParallelI/O

InputCircuits

InputCircuits

Test Signal

Test Signal

Failsafe

Failsafe

Failsafe

Sensors

*

*

*

VATC Simplified Block Diagram


100 %

Test Effectiveness/CoverageInitialization Tests – Performed upon power up



Initialization Test Coverage

SerialI/O

RAM

WatchdogTimer *

ParallelI/O

ROMInterrupt

Controllers

ParallelI/OCPU

SerialI/O

RAMROMInterrupt

Controllers

ParallelI/OCPU

WatchdogTimer *

ParallelI/O

InputCircuits

InputCircuits

Test Signal

Test Signal

Failsafe

Failsafe

Failsafe

Sensors

*

*

*


100 %

100 %

Test Effectiveness/CoverageInitialization Tests – Performed upon power up






Operational Test Coverage

SerialI/O

RAM

WatchdogTimer *

ParallelI/O

ROMInterrupt

Controllers

ParallelI/OCPU

SerialI/O

RAMROMInterrupt

Controllers

ParallelI/OCPU

WatchdogTimer *

ParallelI/O

InputCircuits

InputCircuits

Test Signal

Test Signal

Failsafe

Failsafe

Failsafe

Sensors

*

*

*


100 %

100 %

100 %

Test Effectiveness/Coverage







Testing with special test equipment of vitaland non-vital hardware components


Safety Certification Test Coverage

SerialI/O

RAM

WatchdogTimer *

ParallelI/O

ROMInterrupt

Controllers

ParallelI/OCPU

SerialI/O

RAMROMInterrupt

Controllers

ParallelI/OCPU

WatchdogTimer *

ParallelI/O

InputCircuits

InputCircuits

Test Signal

Test Signal

Failsafe

Failsafe

Failsafe

Sensors

*

*

*


Hardware Testing

Input Circuits

Serial I/O

Parallel Inputs

CPU

Interrupt Controller

Watchdog Timer

Exercised with periodic test signal

Use of checksums and CRC codes


Check-in/out tests and watchdog timer

Failsafe – no testing required

Instruction set tests and data cross-comparison

Parallel Outputs Periodic test with redundant feed back


Hardware Testing (cont’d)

ROM Use of checksums

RAM Double storage of vital data

Disabling of interrupts (except for NMI)during execution of vital modules


No program calls from vital modules

Check-in/check out test





SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



SAC for S/W Design Errors-Westinghouse Claim-By modular design and a progressively

structured review process, all prudent and practical steps have been taken to eliminate software errors

THEREFORE

Probability of the existence of an unsafe software design error is assumed to be zero

SAME


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



SAC for S/W Translation Errors

Use of Assembly Language to reduce the risk of COTS Development Tool bugs

Coding Standards developed and used during translation from Pseudo Code to Executable Code

Unit testing on all vital software modulesSAME


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



SAC for Implementation Errors

Double Storage of Vital Program Code

Checksum stored with program code

MODIFIED

SAME


Double Storage of Vital Program Code-Baseline

Faulty EPROM hardware AND/OR faulty software tool will probably not be faultyin the same way at two different memory locations.

If glitch occurs while “burning” one locations, it is unlikely that the same glitch willoccur while burning the same information in another location

Vital Program

Non-Vital Program

Copy 1

Copy 2

One Copy

EPROM


Double Assembly of Program Code - Modified Release -

Vital And

Non-VitalProgram

Code

Copy 1

Copy 2

COMPARE

COTS SET 1

COTS SET 2

USE

HexCode

HexCode


Checksum on Program Code

Address Data

00010000

000200030004000500060007000800090010001100120013001400150016

XXXX

0010 11110110 11000101 1100

0001 1010

0111 0000

0010 11110110 11000101 11000001 1010

0111 0000

0000 0000

0101 01011111 0000

0101 01011111 1111

1111 00000001 1101

0011 1111

ADD

CHECKSUM

Then during programoperation, sum contentsof memory and comparewith checksum.

Any mismatch interpretedas program code fault.

EPROM

SAME


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN

ERROR

TRANSLATIONERROR


RUN TIMEERROR

EPROMBURNER

Intel8086



Run Time Errors

PROCESSING HARDWARE

Computer System

InputHardware

sensors antennas receivers

etc.

OutputHardware

driversamplifiers

transmittersEtc.

Vital Software

Non-Vital Software

Software design must protect against hazardous operation in the presence of:

Hardware failures

Corrupt processor behavior caused by errors in the non-vital software

SAME


Checked Redundancy SAC to Protect Against Run-Time Errors

Redundant implementation of critical functions.

Detected discrepancies blocked from output

System shut down if failure persists

Probabilities calculated to determine MTBH


Implementation

PeriodicallyChecked


Compare

SAME



Implementation

PeriodicallyChecked


Compare

Channel IndependenceA and B must have no common mode failure mechanisms

Channel IntegrityA and B must be correctly implemented

Channel InspectionA and B must be tested periodically

The Three I’s of a Successful Checked Redundant Design

SAME



Implementation

PeriodicallyChecked


Compare




Pus is very much a function of the check interval and the effectiveness of the check

SAME


Test Intervals FrequenciesInitialization Tests – Performed upon power up



TIMELINE

Safety Certification – every 365 days

Initialization - every 60 hrs

Operational–10 s to 1.2 hrs

SAME


Test Methodology







Testing with special test equipment

SAME


SerialI/O

RAM

WatchdogTimer *

ParallelI/O

ROMInterrupt

Controllers

ParallelI/O

CPU

SerialI/O

RAMROMInterrupt

Controllers

ParallelI/O

CPU

WatchdogTimer *

ParallelI/O

InputCircuits

InputCircuits

Test Signal

Test Signal

Failsafe

Failsafe

Failsafe

Sensors

*

*

*

VATC Simplified Block Diagram

SAME


Hardware Testing

Input Circuits

Serial I/O

Parallel Inputs

CPU

Interrupt Controller

Watchdog Timer


Use of checksums and CRC codes


Check-in/out tests and watchdog timer

Failsafe – no testing required

Instruction set tests and data cross-comparison

Parallel Outputs Periodic test with redundant feed backSAME


Hardware Testing (cont’d)

ROM Use of checksums

RAM Double storage of vital data


No program calls from vital modules

Check-in/check out test

SAME

MODIFIED

SAME





Comparing the Baseline to the Modified SAC’sProtection from:

Design Error Same NA

Translation Error Same NA

Implementation Error Modified Baseline: doubleStorage of vital code

Modified: double Compiling of all code

Run Time Error Modified RAM data Protection modified

Comparison Difference





Summary


Safety Confirmation

VALIDATIONWas it designedto do the right thing?

VERIFICATIONWas the design translated/implemented correctly?

High Level Requirements

Software RequirementsPseudo Code

Hex CodeEPROM


MMF Software Release


AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus


MMF Software V&V Objectives


AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus




AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus




AATCRadio/ATIC

VehicleDoor Relays


BaselineFunctions


NewFunctions


AATC SpeedCommands

VehicleStatus

Hardware Testing


MMF Release

Unsafe Condition

Baseline functionsImplemented incorrectly

New AATC FunctionsImplemented incorrectly

Functional logicImplementedincorrectly

H/W CheckingCode implemented

incorrectly

Baseline functiionsModified incorrectly

Baseline functionsNot isolated from

New functions

CMF Release

High Level Fault Tree Structure


Element 4 and 5 Documentation

Element 4 VATC Subsystem V&V PlanVATC Modifications V&V Report

Element 5QA Plan and ReportSafety Assurance ConceptFault Tree AnalysisSafety Assurance Concept Implementation

AnalysisFMEA’sAudit Reports


AATC Safety Analysis/Reports

SAFETY ASSURANCE CONCEPTS (SAC)

Document explaining what features of the design ensures that hazards are adequately mitigated.

SAFETY ASSURANCE CONCEPTS IMPLEMENTATION (SACI)

Analysis confirming that SAC are implemented

FAULT TREE ANALYSIS (FTA)

Graphic analysis identifying which functions are safety critical

FAILURE MODES AND EFFECTS ANALYSIS (FMEA)

Circuit analysis confirming failsafe behavior of critical circuits

S/W V&V REPORT

Requirements tracing, V&V records


Some Key Westinghouse Analysis Documents

Baseline Fault Tree Analysis

Identifies combinationsof failures that can leadto unsafe operation

System Hazard Analysis

(SHA)

Computes probabilityof unsafe combinationoccurring and ultimatelythe system MTBH


The S/W Analysis Process

BaselineFTA

MMF FTA


BART ATCUnsafe

Condition

Train speed greater thanCommanded speed without

Adequate braking

Doors commanded to openIn unsafe manner

External BRK3Failure

Failure to respondTo brake request Generated by pro-cessing kernel

Brake requestNot generated byProcessing kernel

Baseline Fault Tree Structure


BART ATCUnsafe

Condition

Train speed greater thanCommanded speed without

Adequate braking

Doors commanded to openIn unsafe manner

External BRK3Failure

Failure to respondTo brake request Generated by pro-cessingkernel

Brake requestNot generated byProcessing kernel

BART ATCUnsafe

Condition

New AATCFunctions

Tree Structure Modified for AATC


Baseline Construct Representing Checked Redundancy

Hazard Event

Hazard Event HazardEvent

LeaderHazardous

Event

FollowerHazardous

Event


Fault Conditions for Modified Release

Hazard Event

Hazard EventMechanism requiringLeader & Follower

modified

LeaderHazardous

Event

FollowerHazardous

Event

Hazard Event

UnintendedOperation of S/WGenerates Hazard

Z.1



BaselineFTA

MMF FTA

List of BasicEvents

SACI

MTBHSHA

SAC

List of Assumptions

List of Requirements

Corrected MTBH

SMRS

AssemblyCode

PseudoCode

FMEA

List of BasicEvents



BaselineFTA

MMF FTA

List of BasicEvents SACI

SHA MTBH

List of Assumptions

List of Requirements

SMRS

AssemblyCode

PseudoCode

Corrected MTBH

SAC

FMEA

ATTACHMENT 5.4ATTACHMENT 5.5, SECTIONS 3 & 4ATTACHMENT 5.5, APP. B

ATTACHMENTS 5.6 & 5.7


Fault Tree “Z-Branch” Analysis


What is the “Z-Branch”

Certain aspects of all vital computer and software implementations require V&V regardless of the specific application.

These common “problem areas” include:– RAM/ROM– Program Pointer– The Stack– Interrupts– Pointer and Index Variables

These are describedin the VATC Safety Assurance Concepts document, Section 2.3.5


What are these “Problem Areas?”

Computer

Inputs

Outputs

VATC

1) Things that can happen when the computer is running

Divided into Two Types:


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

ROM

Vital Non-Vital

EPROMBURNER

What are these “Problem Areas?”

2) Things that can happen during software development


What are these problem areas?

RAM and ROM Read Only Memory (ROM) contains permanent,

non-changing data Random Access Memory (RAM) contains dynamic,

changing data What can go wrong:

– Hardware failures can corrupt memory contents

– Software errors can corrupt memory contents

EPROM


What can go wrong? - Hardware glitch can alter the program pointer - Software errors can corrupt the pointer


The Program Pointer Keeps track of which software instruction to

execute next

Module D Module B

Module A

Module C



The Stack Stack Pointer

– Determines where next data will be read from or written to stack memory

Stack Memory Contents– An area in RAM

used to temporarily store data

Stack Pointer

Stack Memory

(256 bytes) 1

2

3

4

5

256

255

:

:



The Stack What can go wrong?

– Hardware glitch can corrupt the stack pointer or the contents of stack memory

– Software error can corrupt the stack pointer or the contents of stack memory



Interrupt Processing

Module F

Module H

Module E

Module G

Module D Module B

Module A

Module C

Main loop with interrupting processes




Module D Module B

Module A

Module C

Main Loop

The Main Loop executes continuously as fast as possible




Module D Module B

Module A

Module C

Module F

Module H

Module E

Module G

Modules E & F stop the main loop from processing until they are completed




Module F

Module H

Module E

Module G

Module D Module B

Module A

Module C

What can go wrong?– Software errors can alter interrupt frequency– Software errors can prevent interrupts from being

serviced in a timely manner (latency)



Pointers and Indexes Allows access to any part of memory

:

:

125

124

123

122

121

123

:

:

:

:

16

7

12

125

89

:

:

ADDRESS CONTENTS

12

:

:

14

13

12

11

10

:

:

:

:

105

13

250

127

65

:

:

ADDRESS CONTENTS

250

POINTERVARIABLE

RETURNVALUE

MEMORY MEMORY


What have we done to identify and protect these “Problem

Areas?”


UnintendedSoftware O peration

Causes Hazard

S/W Design ErrorResults in

HazardousO peration

F irm wareIm plem entationError Results in

Hazard

Vita l Logic M odifiedand Introduces

Hazard

Non-V ita l Log icCorrupts V ita lO peration and

Causes Hazard

O ther V ita l Log icAffects Execution or

Results o f V ita lLogic

Protection fromCorruption of

Variable Data notProvided

No Protectionfrom

Corruption ofS tack

13

Vita l Logic forFunctionsM odified

Incorrectly8

M odificationsAffect

Execution ofV ita l Log ic

9

M odificationsAffect Resultsof V ita l Log ic

10

DataCorrupted

DuringInterrupt

Processing11

DataCorruptedBetween

Interrupt Calls12

Error DuringAssem bly, L ink orConversion to Hex

Error DuringBurning of M aster

EPRO M

NoM itigation forFault DuringCopying of

M aster18

NoM itigation for

Error inSoftware

Tools14

NoM itigation forError at T im eof T ranslation

15

NoM itigation for

Fault inEPRO MBurner

16

NoM itigation forError DuringBurning of

M asterEPRO M

17

Part 1 Part 2

VATC FTA Z-Branch



HazardousO peration


Hazard

Non-V ita l LogicCorrupts V ita lO peration and

Causes Hazard

O ther V ita l LogicAffects Execution or




No Protectionfrom

Corruption ofStack

13


Incorrectly8


Execution ofV ita l Logic

9

M odificationsAffect Resultsof V ita l Logic

10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #8Vital function implemented incorrectly.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

WE VERIFIED THAT:

-The function was identified as “vital” (SAC Implementation Analysis, Appendix C.3)

-The function was properly unit tested (SAC Implementation Analysis, Appendix C.3, V&V Report, Section 2)

-A code review was performed on the function (SAC Implementation Analysis, Appendix C.3, and V&V Report, Section 9)



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

Identification of Vital Software Modules

- The SAC Implementation Analysis examined each basic event of the fault tree and identified where in the VATC software associated functions were implemented

- These vital software modules are listed in Appendix C.3 of the SAC Implementation Analysis



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1Confirmation of Unit Testing

- Inspection of software module unit test results verified that modified vital modules were properly tested

- Unit testing isolates each module and confirms that the proper outputs result from given inputs

- Unit test results are in the V&V Report, Section 2



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

Confirmation of Code Review

- Inspection of code review results verified that modified vital modules were properly reviewed

- Code reviews ensure that vital modules are implemented correctly

- Code review results are in the V&V Report, Section 9



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #9Software error in one vital function prevents or alters execution of another vital function.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

WE VERIFIED THAT:

-The vital function contains check-in and check-out runtime tests (SAC Implementation Analysis, Appendix C.3)

-Interrupt timing and latency tests were performed on the interrupt string (SAC Implementation Analysis, Appendix D)


FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

Check-In & Check-Out Runtime Tests

-Check-In/Out runtime tests detect out-of-sequence program execution

- Inspection of software source code verified that vital modules include these tests



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

Interrupt Timing Tests

- Confirm correct frequency

- Confirm correct latency (interrupt service delay)



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #10Software error in one vital function alters the results of another vital function.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


WE VERIFIED THAT:

-Only the correct functions modify the contents of critical variables (SAC Implementation Analysis, Appendix C.4)

-All memory writes using pointers or indexes are within their defined array bounds (SAC Implementation Analysis, Appendix C.5)

-All stack operations are proper (SAC Implementation Analysis, Appendix C.6)

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1There are only three ways a software module can alter the results of another software module:

- Writing to memory using a critical variable name

- Writing to memory using a pointer or index

- Altering the stack in an improper way



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1 Only Correct Functions Update Critical Variables

- Critical variables were identified as any variable used by a vital module

- All software modules were searched for each identified critical variable name

- Each occurrence of a critical variable was examined to ensure that its usage was correct as per design



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1Pointer and Index Variables are Used Correctly when Writing to Memory

- All memory writes using pointers or indexes were identified

- Simple software was inspected to ensure memory writes using pointers/indexes are always within proper bounds

- Complex software had runtime bounds check added just before memory write occurs



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12


FTA Z-Branch

Part 1

Stack Operations are Correct

- Runtime stack pointer range check was implemented

- All software instructions which alter the stack (change the stack pointer, push data on the stack, pop data off the stack) were identified and examined to ensure that they were correctly implemented



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #11Software error causes corruption of critical variable data during interrupt processing.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #11Software error causes corruption of critical variable data during interrupt processing.

FTA Z-Branch

Part 1

WE VERIFIED THAT:

-Only the correct functions modify the contents of critical variables (SAC Implementation Analysis, Appendix C.4) *SAME*

-All memory writes using pointers or indexes are within their defined array bounds (SAC Implementation Analysis, Appendix C.5) *SAME*

-All stack operations are proper (SAC Implementation Analysis, Appendix C.6) *SAME*



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #12Software error causes corruption of critical variable data between interrupt calls.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #12Software error causes corruption of critical variable data between interrupt calls.

FTA Z-Branch

Part 1

WE VERIFIED THAT:

-Only the correct functions modify the contents of critical variables (SAC Implementation Analysis, Appendix C.4) *SAME*

-All memory writes using pointers or indexes are within their defined array bounds (SAC Implementation Analysis, Appendix C.5) *SAME*




HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #13Software error causes corruption of stack, which causes hazard.

FTA Z-Branch

Part 1



HazardousO peration


Hazard


Causes Hazard





No Protectionfrom

Corruption ofStack

13


Incorrectly8



9


10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12

Basic Event #13Software error causes corruption of stack, which causes hazard.

WE VERIFIED THAT:

-Added runtime stack pointer range check (SAC Implementation Analysis, Appendix C.6) *SAME*


FTA Z-Branch

Part 1


UnintendedSoftware O peration

Causes Hazard


HazardousO peration

F irm wareIm plem entationError Results in

Hazard


Hazard

Non-V ita l Log icCorrupts V ita lO peration and

Causes Hazard

O ther V ita l Log icAffects Execution or




No Protectionfrom

Corruption ofS tack

13


Incorrectly8


Execution ofV ita l Log ic

9

M odificationsAffect Resultsof V ita l Log ic

10

DataCorrupted

DuringInterrupt

Processing11


Interrupt Calls12



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Part 1 Part 2

VATC FTA Z-Branch


Firm wareIm plem entationError Results in

Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Basic Event #14Assembler, linker or other conversion process introduces error in executable code.

FTA Z-Branch

Part 2

Buggy assembler/linker

tool suite



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17


FTA Z-Branch

Part 2


tool suite

WE VERIFIED THAT:

-Two independent assembler/linker tool suites were used to generate two executable files, and that these files match (SAC Implementation Analysis, Appendix E)



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Use Two Independent Tool Suites:

- Low probability that two different compiler tools contain the same error


FTA Z-Branch

Part 2


tool suite



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Basic Event #15A computer glitch during assembly, linking or other conversion process introduces error in executable code.

FTA Z-Branch

Part 2

Glitch during assembly process



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

WE VERIFIED THAT:

-The process of assembly and linking is performed twice, and that the two resulting executable files match (SAC Implementation Analysis, Appendix E)


FTA Z-Branch

Part 2




Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Build the Executable Twice:

- Low probability that a random computer glitch could produce the same error during two different build operations


FTA Z-Branch

Part 2




Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Basic Event #16Executable code becomes corrupted by faulty EPROM burner.

FTA Z-Branch

Part 2

Buggy EPROM burner



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

WE VERIFIED THAT:

-Runtime EPROM checksum is in place (SAC Implementation Analysis, Appendix E)


FTA Z-Branch

Part 2

Buggy EPROM burner



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Runtime EPROM Check Code Protects Software After Build Process:

- Once the translation process is complete, it is protected by a checksum that can detect any later changes


FTA Z-Branch

Part 2

Buggy EPROM burner



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Basic Event #17Executable code is not correctly burned into master EPROM.

FTA Z-Branch

Part 2

Glitch during EPROM burning



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

WE VERIFIED THAT:

-Runtime EPROM checksum is in place (SAC Implementation Analysis, Appendix E) *SAME*

Basic Event #17Executable code is not correctly burned into master EPROM.

FTA Z-Branch

Part 2

Glitch during EPROM burning



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

Basic Event #18Copies of master EPROM do not exactly match the master EPROM.

FTA Z-Branch

Part 2

Failed EPROM



Hazard



EPRO M


M aster18

NoM itigation for

Error inSoftware

Tools14


15

NoM itigation for


16


M asterEPRO M

17

WE VERIFIED THAT:

-Runtime EPROM checksum is in place (SAC Implementation Analysis, Appendix E) *SAME*

Basic Event #18Copies of master EPROM do not exactly match the master EPROM.

FTA Z-Branch

Part 2

Failed EPROM


VATC SubsystemDesign Verification


VATC MMF Verification Activities

Design Review Software Module Desk Checks Group Walk-Through Code Review Traceability Matrix Testing Open Issues Closeout


Section 1 – Report Organization Section 2 – Unit Testing ReportSection 3 – Lab Integration TestingSection 4 – ERS Integration TestingSection 5 – Hayward Test Track TestingSection 6 – Mainline Track TestingSection 7 – Traceability MatrixSection 8 – Software Module Desk CheckSection 9 – Software Module Code ReviewsSection 10 - Final Design ReviewSection 11 – Open Issues Closeout

V&V Report Contents


VATC MMF Design Review (V&V Report Section 10)

• Presentation of Software Design to technical staff• Participation of Harmon, Sandia and various BART

Departments• Generation of Action Items List• Closeout of Action Items List


V&V Report Contents



Software Module Desk Checks(V&V Report Section 8)

Specifications

Requirements

Pseudo

Code

Source Code

Generated Report listing discrepancies founded and corrected.

Cross

check

Cross

check



V&V Report Contents


Group Walk-Through Code Review (V&V Report Section 9)

Performed on modified/new vital software modules Verified that proper logic was in place and source code

represented approved design Verified that coding standards and styles guidelines

were followed Checklist provided to ensure completeness in the

evaluation



V&V Report Contents


VATC Traceability Matrix (V&V Report Section 7)

Each AATC System Requirement relating to the VATC is traced to the VATC Design, Code and Test.

SysRSTable

SRSTable

ModuleTable

TestTable

Mapping betweenSysRS and SRS

Mapping betweenSRS and Modules

Mapping betweenSRS and Tests


SYSRS to SRS Report

Each AATC System Requirement relating to the VATC is traced to one or more VATC Subsystem Requirement

SysRSTable

SRSTable

ModuleTable

TestTable





SRS-Software Module Report

Tracing of each VATC Subsystem Requirement to one or more software modules

SysRSTable

SRSTable

ModuleTable

TestTable





SRS-Test Report

Tracing of each VATC Subsystem Requirement to one or more Test

SysRSTable

SRSTable

ModuleTable

TestTable






V&V Report Contents


VATC Unit Testing – MMF (V&V Report Section 2)

Performed on vital modified/new software modules

Test results verified that Min/Max/Zero input values produced expected results

Test results verified that all branches in the module were executed

Software Module

All possible Inputs

All Outputs SafeAll Branches Checked



V&V Report Contents


Laboratory Subsystem Integration Testing - MMF (V&V Report Section 3)

AATU / AATC SIMULATOR(Snooper/Driver)

LCDConsole

Win9x

ControlPanelSwitch Box37 Volts

ATPCradle O

N

RS-232Serial Link,Power, Etc.

5 Volts

POWER SUPPLY

37 Volts 5 Volts

RS-232Serial Link

•Verified functionality of ATP Cradle

•VATC in Monitor Mode only

•Generation of ATP reports

•Decoding of AATC commands



V&V Report Contents


ERS Subsystem Integration Testing – MMF (V&V Report Section 4)

Verified proper functionality of Vehicle ATC using existing Primary Test Equipment.

Main PowerSwitch and

A11-J1ConnectorAccelerometers

AATC SIMULATOR(AATU)

WestinghousePrimary Test Equipment

RS-232Serial Link

Track Circuit Simulation/ Tests AATC Control Tests

ATCCradle

2

ATCCradle

1

ATCCradle

3

Laptop computer

RS-232Serial Link

RS-232Serial Link



V&V Report Contents


Hayward Test Track Testing – MMF (V&V Report Section 5)

Verified VATC never goes under AATC control

Verified proper Automatic Train Operation under existing track circuit system, including:

- Motion Control- Door Operation- Wayside Communications



V&V Report Contents


Mainline Testing – MMF (V&V Report Section 6)

Verified Proper automatic track circuit control operation

Verified IDEN coverboard communications



V&V Report Contents


Open Issues Closeout (V&V Report Section 11)

Open Issues Record – problems reported and corrective actions taken during Design, Implementation, Testing and Analysis of the VATC Subsystem.

Found

Fixed

Problem



15 Non-Vital Open Issues Identified

Memorandum – documents why closure of 15 Non-Vital Issues is NOT necessary prior to MMF Safety Certification



Unit Testing Report

– Lists discrepancies found during Unit Testing

– Cross-references the discrepancy to an item in the Open Issues file (where applicable).



V&V Report Contents





Summary


SUMMARY

Emphasis of the MMF certification has been to demonstrate that baseline functions have not been corrupted– Functional requirements still intact– Hardware checking routines still in place– Proper isolation between new and old functions

Safety requirements for the baseline system maintained for the modified release


SUMMARY (cont’d)

Proper understanding of the safety assurance concepts demonstrated in the SAC document

Safety analysis developed and confirmed vital requirements using a structured and orderly process documented in the FTA and SACI documents


SUMMARY (cont’d)

Complete and proper implementation of the SAC’s confirmed by examination of the potential hazards identified in the FTA Z-Branch and documented in the SACI

Comprehensive tracing from: – requirements to implementation and– requirements to test program

Completed and documented in the V&V Report


SystemRequire-

ments

SubSystem

Require-ments


ments

PseudoCode

HexCode

AssemblyCode

AssemblyCode

ROM

RAM

ROM

Vital Non-Vital

DESIGN TRANSLATION

IMPLEMENTATION

EPROMBURNER

Intel8086

SUMMARY

OPERATION


Conclusion

The VATC MMF software is safe and ready for revenue service!