1

Advanced Engineering Risk Audit Instruments

SYRAS & QMAS

Bob Bea

National Science Foundation RESIN PROJECT

2

Probabilities & Consequences of Failure As IsM itig a tio n M ea su res

Probabilities & Consequences of Failure As Could BeIm p le m en ta tio n P ro gra m s

SYRASN o rm a l ra te s o f m a lfu nc tio ns , P erfo rm a n ce S ha p ing F ac to rs

P ro b ab ilit ie s o f p ro p e r D e te c tio n , A n a lys is , C o rre c tionP ro b ab ilit ie s o f F a ilu re fo r a ll S ys tem C om p o ne n ts , C o rre la tio ns

QM ASA u d it tea m , A u d it tra in ing , A u d it s ch e du ling

S c re en in g a ssessm e n ts, F a cto rs o f C on ce rnG ra d in g s , C o nse q ue n ce s o f 'F a ilu re ', R e m e d ia tio ns

D e fine & C h a ra cte rizeSystem, Perform ance Attributes & Failures

O p e ra to rs , O rg a n iza tion s , E n v iro n m e n tsH a rd w a re , S truc tu res , P ro ce du re s , In te rfa ces

3

Managing Pfi

C onc eptD efin ition

Des ign

C ons truc tion M aintenanc e

Dec ommis s ioning

P resent Future

Reduce L ikelihoods Reduce Eff ec ts Inc rease Detec tion

& Correc tion

Inc reas e

Margins of Quality

P fi

4

Managing Cfi

R es c ue R epa ir

R ec overy R ehabilita tion

Immediate Delayed

Direc t Indirec t On-S ite Off -S ite

C fi

5

Resilience - time & cost to regain Functionality

Time

Fu

nct

ion

alit

y

100 %

rescue

repairrecovery

rehabilitation

Good resililence

Poor resililence

6

Sustainability - Utility during life cycle

Time

Uti

lity

Benefits / Costs

Initiation Improvement

Operation Low Sustainability

High Sustainability

Decay

7

PROACTIVE to prepare

INTERACTIVE to respond

REACTIVE to recover

reduce consequences

increase detection &correction reduce likelihoods

8

An

alyt

ical

met

hod

s

QUALITATIVE'soft' - linguistic variables

'subjective' integration

QUANTITATIVE'hard' - numerical variables

analytical integration

MIXED QUALITATIVE & QUANTITATIVElinguistic variables translated to numerical variables

analytical integration

Low Moderate High

0.00 5.00 10.00

Low

0 5.0 10

Moderate High

2.5 7.5

9Sys

tem

an

alys

is a

pp

roac

h

Step #1: PERFORM A SYSTEM ANALYSIS

define system hardware, software, environments,organizations and individuals

Step #2: PERFORM A PROCESS ANALYSIS

define how things are done: how the system hardware,software, environments, organizations, and individuals

work

Step #3: PERFORM ANALYSES OF THE SYSTEM &PROCESS AS PRESENTLY CONFIGURED

define the likelihood and "quality" achieved

Step #4: PERFORM ANALYSES OF THERECONFIGURED SYSTEM & PROCESS

define the likelihood and additional"quality" that could be achieved

10

Screening analysis strategy

Develop CoarseQualitative Model

of Existing"System"

Identify High LikelihoodSystem Problem and

Consequence Componentsand Potential Mitigation Measures

Develop DetailedQuantitative Models

of High Risk Components

EvaluateHigh Risk

Components

Revise Models andEvaluate Risk Reductions

of Mitigation Measures

Evaluate Costs andBenefits of Mitigation

Measures and Impliment"Most Effective"

11

Objectives of analyses

• NOT prediction

• NOT numbers or analytical elegance

• Detect potential hazards and flaws

• Evaluate effects of mitigations

• Disciplined framework for thinking

• Empower those who have daily responsibilities for quality and reliability

12

Exa

mp

le Q

RA

/PR

A

13

14

Pro

acti

ve -

Rob

ust

nes

s =

dam

age

tole

ran

ce

Magnitude (Y) of Type (X) of Human Error

Pfs|e1.0

0.0

error tolerant

error intolerant

Intensity of damage or defects

defect tolerant

defect in tolerant

15

Durability

QA / QC LifecycleTask Organization

QA / QC Task Organization

InfluencingFactors

Base Rates

Task 1

InfluencingFactors

Base Rates

Task 2

InfluencingFactors

Base Rates

Task 3

DesignPhase

ConstructionPhase

OperationsPhase

QA / QC Task Organization

InfluencingFactors

Base Rates

Task 1

InfluencingFactors

Base Rates

Task 2

InfluencingFactors

Base Rates

Task 3

MaintanencePhase

Safety Compatibility Serviceability

QualityAttributesSYRAS©

16

Failures of QA/QC

Independent&

DependentErrors

Errors

•lack of expertise•resource deficiencies•excessive authority gradients•communication malfunctions•management breakdowns•rejection of information•violations

Correction

Detection

Analysis

17

P[F] = P[FI|E] P[E]+P[FI|E]P[E]+P[FE|E]P[E]

due to intrinsic causes with no error

due to intrinsic causes with error

due to extrinsic causes with error

18

19

20

21

221 E-6

1 E-5

1 E-4

1 E-3

1 E-2

1 E-1

1unfamilar taskperformed withspeed

simple taskperformd withspeed

change systemwith procedureswith checking

routine taskstrained, motivated

respond to system commandswith supervisory system

change system statewithout procedureswithout checking

routine taskperformed withspeed or divertedattention

23

1 1E11E-11E-21E-3 1E2 1E3SYRAS PSF

Extr

em

e ne

gati

ve

influen

ce

Mode

rate

neg

ative

influen

ce

Min

or

neg

ativ

e in

fluenc

e

Neglig

ible

influenc

e

Extr

eme p

ositiv

e in

fluenc

e

Moder

ate p

osit

ive

influe

nce

Min

or p

osi

tive

influen

ce

SMAS Grade 1 2 3 4 5 6 7

Outs

tandin

g

excee

ds

all s

tand

ard

s an

d r

equirem

ent

s

Goo

d, av

era

ge

exc

eed

s a

ll st

and

ards

and

req

uire

ment

s

Very

poor

doe

s not

mee

t an

y s

tand

ard

s or re

quirem

ent

s

Exc

elle

nt

Ver

y g

ood

Bel

ow

avera

ge

Poo

r

24

QMAS©

• Assessment instrument

• Assessment protocol

• Assessment team development

• Follow-up

25

SMAS Main Menu

This is the starting point of the SMAS Data Management Tool.

Pre-Assessment Data - This way to Pre-Assessment Data Input Menu

Phase 1 Data - This way to Phase 1 analysis Data Input Menu

Phase 2 Data - This way to Phase 2 analysis Data Input Menu

FOCÕs / Scenarios - This way to Factors of Concern / ScenarioÕs Menu

Reports - This way to Reports

Utilities - This way to Import, Export, Delete Utilities

System Exit - This way to Exit SMAS and close MS Access.

26

QMAS components

OperatingTeamsFactors

OrganizationalFactors

ProceduralFactors

HardwareFactors

StructuralFactors

InterfacesFactors

EnvironmentalFactors

27

QMAS components, factors, attributes

COMPONENTS

1 - operators, 2 - organizations, 3 - procedures,4 - equipment, 5 - structure, 6 - environments,

7 - interfaces

FACTORS - graded

1.0 - operators1.1 - communications, 1.2 - process auditing, 1.3 - safety culture,

1.4 - risk perception 1.5 - emergency preparedness, 1.6 - command & controls,1.7 - training, 1.8 - resources, 1.9 - requesite variety

ATTRIBUTES - reasons for grades

1.1 - communications1.1.1 - same language, 1.1.2 - same vocabulary, 1.1.3 - established forms,

1.1.4 - clarity, 1.1.5 - concise, 1.1.6 - timely, 1.1.7 - appropriate amount1.1.8 - feedback, 1.1.9 - no significant barriers

28

QMAS factors

29

QM

AS

gra

din

gs

1

2

3

4

5

6

7Very poor, does notmeet any standards

or requirements

Outstanding,exceeds all standards

and requirements

Good, average,meets most standards

and requirements

Poor

Excellent

Very good

Below averagehigh bound

mostprobable

low bound

30

31

QMAS mean results

Operators

Organizations

Procedures

Equipment

Structure

Environments

Interfaces

0 1 2 3 4 5 6 7

Co

mp

on

ents

Grades

Highest mitigation priorities

32

QMAS protocol

Stage #1ONSHORE

Stage #2OFFSHORE

Stage #3ONSHORE

general informationsystem specificinterviews - operating, management

walk downsoperations observationsmaintenance observationsemergency drillsinterviewsanonymous discussions

additional informationadditional interviews

33QM

AS

- c

oars

e q

ual

itat

ive

.

Step #1Select System

for Assessment

history of LOClikelihood of LOCconsequences of LOCschedule for assessment

Step #2Identify

AssessmentTeam

Designated InspectionRepresentatives (DIR's)

Step #3Coarse Qualitative

Assessment

general historysystem detailsoperations observationsprevious hazard studiesmanagement interviewsoperator interviews

Factorsof Concern

(FOC)

standard reports(SEMP, PFEER, ISM, user)rationalle for FOCmitigation measures

Proceed ?implement mitigationsrecord resultsschedule next assessment

no

yes

new

FO

C

34QM

AS

det

aile

d q

ual

itat

ive

.

Step #5Detailed

QualitativeAssessment of

FOC

general historysystem detailsoperations observationsprevious hazard studiesmanagement interviewsoperator interviews

SuggestedMitigationMeasures

mitigation measuresrationalle for suggestionsprojected effects

Proceed ?implement mitigationsrecord resultsschedule next assessment

no

proceed tocoarse quantitative

analyses of FOC

yes

yes

Step #4Develop Scenarios

for each FOC

new

FO

C

35

36

37

QM

AS

ass

esso

rs• QMAS counselor

• QMAS assessors– Facility operators

– Facility management

• QMAS training program– HOF background

– QMAS operations

– QMAS applications

38

QMAS assessors: THE MOST IMPORTANT PART!

• Experienced– Facility– Process auditing

• Respected

• Integrity

• Motivated

• Observant

• Thoughtful, insightful

39

QMAS applications