Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
OLF pres JEV 12122011 1Risk management research and developmentPreventorUsumacinta (Pemex), nov 2007
Seminar om KPI – Anleggsintegritet/prosess-sikkerhet
Lagging and Leading Key Performance Indicators Definitions
How are they are used within our industry
Jan Erik Vinnem, Preventor AS12.12.2011
OLF pres JEV 12122011 2Risk management research and developmentPreventor
Overview
• Background• Terminology• Alternative perspectives• Criteria for good indicators• Challenges• Recommendations
OLF pres JEV 12122011 3Risk management research and developmentPreventor
Brief history of indicators
• Traditionally– Indicators based on occupational injuries– Was proven disastrously misleading by Texas City
explosion• Major hazard risk indicators have received
significant attention after Texas City– Lagging indicators appear to be the main focus in the
process industries• Also the starting point for RNNP in 1999/2000
– Leading & lagging indicators• Total E&P Norge started with major hazard
indicators in 199722
30
21 21
17 17
11 10
18
11 11 1011 11
7
0
5
10
15
20
25
30
35
40
9702 9703 9704 9801 9802 9803 9804 9901 9902 9903 9904 0001 0002 0003 0004
Year/Quarter
Rel
ativ
e ris
k
Overall major hazard risk indicatorfor Frigg Central Complex
OLF pres JEV 12122011 4Risk management research and developmentPreventor
Why special emphasis on major hazards?• Rare occurrences (0 fat. after ’85)• For individual installations in NCS
– Worst: 1–2 major accident precursors per year– Best: many years between precursor events
• Causal mechanisms different– Example: is number of leaks < 0.1 kg/s a good
indicator for leaks with escalation potential?• Leaks >0.1 kg/s has escalation potential• Causes of minor leaks often different from >0.1 kg/s
– Root causes may have more in common
OLF pres JEV 12122011 5Risk management research and developmentPreventor
Terminology confusion?
• KPI; Key Performance Indicator• Risk indicator• Safety indicator• Major hazard risk indicator
• Asset integrity• Process safety• Major hazard risk
• Indicators– Leading, lagging, proactive, barrier, precursor, output, input
• Focus on definitions and subtle differences may be interesting academically, but rarely very fruitful!
Focus in presentation:
• Major hazard risk is the focus area
OLF pres JEV 12122011 6Risk management research and developmentPreventor
-1,0
-0,5
0,0
0,5
1,0
1,5
2005 2006 2007 2008
Rel
ativ
e ov
eral
l bar
rier
avai
labi
lity
Installation 1Installation 2
Important distinction
• On which level are indicators considered?– National level (RNNP)
• National trends, challenges, focus areas, effect of attention, etc.
– Company level (variable)• Benchmarking between companies
– Installation level• Benchmarking between installations• Trends on installations• Problem areas on some installations
0,00,1
0,2
0,3
0,4
0,50,6
0,7
0,80,9
1,0
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 Int01-07
Year
Leak
s pe
r ins
talla
tion
year
s
0
0,1
0,2
0,3
0,4
1 2 3 4 5 6 7 8 9 10 11
Operator
Leak
s pe
r ins
talla
tion
year
> 1 kg/s
0,1 - 1 kg/s
OLF pres JEV 12122011 7Risk management research and developmentPreventor
Leading vs Lagging indictors – is there a problem?
• Recent statement:– ”I consider the definitions to be rather vague & unclear,
especially when some lagging indicators also may be leading indicators.
– Some companies consider HC leaks as lagging indicators, (no of leaks), but may also be considered a leading indicator for fire and explosion.”
• Safety Science– Special issue (2009) Process safety indicators (≈10 papers)– Safety Science (2010)
• Vinnem, JE. Risk indicators for major hazards on offshore installations, 48, 770-787
OLF pres JEV 12122011 8Risk management research and developmentPreventor
Leading indicators
• Kjellén:– Leading safety performance indicator is…….
an indicator that changes before the actual risk level has changed
– Consistent with what is commonly used in economy, but significantly different from what other researchers in safety use
– What is now typically called ‘leading indicators’ earlier was called ‘proactive indicators’
OLF pres JEV 12122011 9Risk management research and developmentPreventor
Implications
• Indicator based on precursor event occurrences are lagging indicators, according to Kjellén’s definition– Risk level has changed, at least temporarily,
when a precursor event occurs• What constitutes a leading indicator appears
to be more controversial
OLF pres JEV 12122011 10Risk management research and developmentPreventor
Further confusion?
• Prof Ben Ale– No indicator for which the values are based on
observations over time can be regarded as leading indicators
• According to this interpretation, none of the indicators used in RNNP would be called ‘leading’
OLF pres JEV 12122011 11Risk management research and developmentPreventor
Leading & lagging indicators
• Relativistic approach (Prof Andrew Hopkins)– A specific indicator could be lagging in relation
to one objective, while leading in relation to another
– Example:• Failure at testing is lagging in relation to the
performance of the individual barriers, while leadingin relation to the performance of the overall process safety management system
OLF pres JEV 12122011 12Risk management research and developmentPreventor
Leading & lagging indicators
• Absolutist approach (Hopkins)– Bowtie based
• All indicators to the left of the ‘knot’ leading• All indicators to the right of the ‘knot’ lagging
– Is all ambiguity thus removed?– Certainly not!– Where is the ‘knot’ located in the sequence:
• Error in flange mounting after PSV replacement• Verification not performed due to ‘silent deviation’• Gas leak• Ignition• Failure of active fire protection• Escalation to neighbour module• Escape and evacuation
OLF pres JEV 12122011 13Risk management research and developmentPreventor
Leading & lagging indicators
• Hopkins– Bowtie is not a good way to
distinguish between leading & lagging indicators
– Relativist approach is not good either
– Alternative classifications needed
OLF pres JEV 12122011 14Risk management research and developmentPreventor
Alternative perspective – lead vs. lag indicator
• Alternative perspective (HSE)– Input – Throughput – Output – Outcome indicators
• Input– Human or organization error leading to gas leak
• Throughput– Culture affecting the chance that errors are not detected
• Output– Failure of fire detector in test
• Outcome– Failure of fire detector to detect real fire Lagging indicator
OLF pres JEV 12122011 15Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Criteria should be strictly tailored to major hazard risk
• Major hazards are special in some senses– Reference made to arguments about frequency– Return period between occurrences of major
hazard precursors• Typical offshore employee• In the range once per 10–15 years
OLF pres JEV 12122011 16Risk management research and developmentPreventor
Avoid “garbage in – garbage out”
• Data basis quality is sometimes a challenge– Unambiguity of reporting criteria
• HC leaks– RNNP: > 0.1 kg/s– HSE: no lower limit
• System scope– Process systems– Fuel gas system– Well systems– What about gas injection system, lift gas system???
– Need to focus continuously on improvements• QC, QC, QC, QC……..
OLF pres JEV 12122011 17Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Combination of lagging and leading indicators• Easily observable performance• Intuitive indicators• Not require complex calculations• Not be influenced by campaigns that give
conflicting signals• Reflect hazard mechanisms• Sensitive to change• Show trends• Robust to manipulation• Validity for major hazard risk Vinnem, Safety Science (2010)
OLF pres JEV 12122011 18Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Combination of lagging and leading indicators– Leading often preferred over lagging– Both should be used, for different needs
• Attention more focused if number of indicators is kept at a reasonable level– Max 20–25?
OLF pres JEV 12122011 19Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Easily observable performance– Measurable by applying a recognized data collection
method and scale of measurement– Usually expressed on a ratio scale of measurement
• Hydrocarbon leaks (>0.1 kg/s) per installation year• Hydrocarbon leaks per 1 million manhours• Reporting reliability may be challenge, such as
– Unclear basis for calculation of rate (kg/s)– Both leak & injury– Both leak & environmental spill
– Limiting the data collection to those events with medium and large severity will usually improve the reliability significantly
OLF pres JEV 12122011 20Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Intuitive indicators– What is measured is considered intuitively by
the workforce to be important for the prevention of major accidents
– Dubious intuitivism• Number of years in the same position (e.g. as
process operator) as an organizational major hazard risk indicator for frequency of HC leaks
OLF pres JEV 12122011 21Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Not require complex calculations– Major hazard risk indicators should not require
complex calculations (intuitiveness)– If the number of observations goes down, it
should correspond to an improvement– If very complex calculations are required, the
confidence may be lost
OLF pres JEV 12122011 22Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Not be influenced by campaigns that give conflicting signals– Psychological and organizational reasons could
in many cases result in a too low reporting– Example
• Think of an organizational incentive structure where absence of incidents is rewarded (for instance by a bonus scheme)
• May result in some incidents not reported, as the incentive structure is interpreted as rewarding the absence of reported incidents
OLF pres JEV 12122011 23Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Sensitive to change• Show trends
– Major hazard risk indicator should allow for early warning by capturing changes in a socio-technical system that have significant effects on accident risks
– “Good" set of indicators reflects changes in risk as well as where improvement areas
– Number of observations (incidents, barrier faults, etc) should for lowest level systems be in order of a dozen or so per period (year?)
OLF pres JEV 12122011 24Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Robust to manipulation– The major hazard risk indicators should not
allow the organization to “look good” by for example changing reporting behaviour, rather than making the necessary basic changes that reduce accident risk
OLF pres JEV 12122011 25Risk management research and developmentPreventor
Criteria for good indicators – major hazard risk
• Validity for major hazard risk– Validity is a critical aspect for goodness of indicators
• Is the indicator a valid indicator for the major accident risk? • Is indicator actually measuring what we intend to measure?
– Consider for example the indicator defined by the number of lost time injuries
• Clearly, this indicator says something about accident risk• Accident risk is more than the number of lost time injuries, so
we cannot use just this indicator to conclude on development in the accident risk level as a whole
– Validity of a statement concerning the major accident risk based on observations of the injury rate only, would thus in most cases be low
OLF pres JEV 12122011 26Risk management research and developmentPreventor
Recommended approach – major accident risk
• Abandon– Terminology: Lagging & leading indicators
• To be preferred– Precursor based indicators– Barrier (proactive) indicators
• Technical barrier elements• Organizational barrier elements
– Still observing the proposed criteria for good indicators
0
5
10
15
20
25
30
35
40
45
50
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Num
ber of leaks
>10 kg/s
1‐10 kg/s
0.1‐1 kg/s
0,000
0,005
0,010
0,015
0,020
0,025
0,030
0,035
Firedetection
Gasdetection
Total Closure test Leak test Total Closure test Leak test DHSV BDV PSV BOP Deluge valve Fire pumpstart
Riser ESDV Wing & Master valve
Ave
rage
frac
tion of
faults
0
1 000
2 000
3 000
4 000
5 000
6 000
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536
Num
ber o
f hou
rs backlog
PM backlog, hours HES critical PM backlog, hours other equipement
OLF pres JEV 12122011 27Risk management research and developmentPreventor
Recommended approach – major accident risk
• Precursor based indicators– Will usually function only on a national level– Installation level
• Too infrequent to provide useful feedback• Should after all be used in order to maintain
awareness & focus
0
5
10
15
20
25
30
35
40
45
50
19961997 1998 1999 20002001 2002 2003 200420052006 2007 2008 20092010
MODU
NUI
Complex
FPU
Fixed production
DFU5 Ships on collisioncourse
OLF pres JEV 12122011 28Risk management research and developmentPreventor
Recommended approach – major accident risk
• Barrier indicators– Important for major hazards in particular
• to focus on indicators that change before the actual risk level has changed
– Will usually function on all levels• National level• Company level• Installation level
0,00
0,05
0,10
0,15
0,20
0,25
D 15
F 27
G 139 H 4 J 8 L 2
M 2
N 4
P 20 Q 4
U 13
V 2
W 6 Y 2
Ø 2
AA 35
AB 27
AC 15
AD 6
AE 46
AF 15
3AH 4
AI 30
AJ 1
2AK 3
AL 11
AM 39
AN14
0AP 8
AQ 18
AR 50
AU 39
AV 7
AW 20
AX 4
AY 8
AZ 22
AÆ 21
AØ 4
AÅ 4
BA 30
BB 12
BC 36
BE 4
BF 61
BG 9
BH 62
BJ 28
BK 42
BL 102
BM 28
BQ 12
BU 4
BV 4
BW 20
BX 2
BZ 2
BØ 2
CA 123
CB 6
CC 10
Fractio
n of fa
ults
2010 Average 2002-10
Riser ESDV