Chapter 14 - Audits and assessments

Chapter 14 - Audits and assessmentsPeter Drucker, 1909–2005
Management Elements Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .720
INTRODUCTION Evaluations of risk management programs can take one of two forms. The first is a formal audit in which a management program is measured against an external standard such as
a regulation or corporate benchmark. An audit compares ‘‘what is’’ with ‘‘what should
be’’. It is fundamentally a pass/fail test. For example, paragraph 1910.119(f)(1) of the
OSHA (Occupational Safety & Health Administration) Process Safety Standard states
that:
Proc
Copy
The employer shall develop and implement written operating procedures that
provide clear instructions for safely conducting <an> . Initial startup;
An auditor who is examining a facility’s operating procedures will check that they do
contain instructions to do with initial startups. If they do, then the audit requirement has
been met. If the auditor finds a deficiency between the requirements of the standard and
what is actually taking place then he/she generates a ‘‘finding’’ or a ‘‘gap’’. As with
Process Hazards Analyses (Chapter 3) it then becomes someone else’s responsibility to
turn the findings into recommendations or action items. An auditor’s job is to objectively
uncover deviations from the standards – no more, no less. The auditor is interested
primarily in the letter of the law. The second type of evaluation is less formal. A reviewer provides an opinion as to the
quality of the risk management program. In the case of initial startup procedures, for
example, a reviewer will provide an opinion as to whether or not those procedures will
actually help ensure that the facility starts safely and according to plan. He/she will
develop that opinion by asking questions to do with the level of detail, the writing style
ess Risk and Reliability Management
right 2010, Ian Sutton. Published by Elsevier Inc. All rights reserved. 693
CHAPTER 14 Audits and assessments694
and the clarity of the instructions. Based on the answers to those questions, he/she will
provide an opinion as to the effectiveness of the initial startup procedures. (The terms
Verification and Validation are sometimes used to make the same distinction between
audits and assessments. Verification is concerned with ensuring that a facility meets the
letter of the law or regulation; validation determines whether it is meeting the spirit of the same law.)
It is important to maintain a clear distinction between the roles of auditor and
reviewer. For example, an auditor may report that many of the elements of the standard to
do with operating procedures have not been completed, and that the audit requirements
have not been met. It is natural at this point for the facility management to ask the auditor
to offer an opinion as to the causes of these failings. If the auditor does offer an opinion
then he/she has moved into a reviewer role. An auditor’s opinions and insights may be
extremely valuable, but they are not a part of the formal auditing process. In the same vein, much of the literature to do with auditing stresses the ‘‘team
relationship’’ that should exist between the auditor and the personnel of the facility
being audited. According to this view both auditor and persons being audited work
together to develop ever higher levels of excellence. This rather rosy picture belies the
fact that a formal audit is a structured process in which a facility’s performance is
measured against some pre-defined standard. The auditor helps only by identifying gaps.
How management elects to close those gaps is not the auditor’s concern. Indeed, given
that the identification of gaps or deficiencies may result in penalties being applied or careers being held back, it would be nave to believe that the auditor and the facility
management are always ‘‘on the same team’’. A reviewer, however, is certainly on the
same team as his or her client.
The challenging nature of formal audits does not mean that they cannot lead to
performance improvements, or that the audit is just a paper exercise. Indeed, it has to be
recognized that the fear of looking bad or of penalties is a strong motivator, and that an
audit can generate major performance improvements. (Concern to do with penalties
was in evidence in the early 1990s. Although the value of process safety management principles had long been recognized, it took a regulation from OSHA to force companies
to complete their process safety work.)
FORMAL AUDITS Audits are a fundamental component of any management system, as shown in
Figure 1.6. By comparing actual performance with written standards, audits help
pinpoint the bad news. The standards may be external, such as those promulgated by a government agency or an engineering society, or they may be internal, such as
corporate standards. Regardless of their source, the standards must be written down and
they must be accepted as authoritative by the facility that is being audited.
Table 14.1 shows the OSHA regulatory requirement for auditing process safety
programs in the United States (it is also discussed in Chapter 15). It is representative of
all effective audit protocols in that it contains within itself the following elements:
- Compliance with a defined, external standard;
- Auditor qualifications;
Table 14.1 OSHA PSM Standard for Audits
(1) Employers shall certify that they have evaluated compliance with the provisions of this section at
least every three years to verify that the procedures and practices developed under the standard
are adequate and are being followed.
(2) The compliance audit shall be conducted by at least one person knowledgeable in the process.
(3) A report of the findings of the audit shall be developed.
(4) The employer shall promptly determine and document an appropriate response to each of the
findings of the compliance audit, and document that deficiencies have been corrected.
(5) Employers shall retain the two (2) most recent compliance audit reports.
Formal audits 695
- Documentation.
The audit and its report will generally be directed toward the facility’s line managers.
They are the ones who provide information to the audit team and who will have the
responsibility of addressing the audit’s findings. Senior managers are not likely to read
routine audits in depth. They rely on their line managers to accept or reject the findings and then to implement the subsequent recommendations. Instead, senior managers are
more interested in understanding how the audit findings can help them improve overall
management systems. They also want to know if any major liability or safety issues have
been identified.
The frequency with which audits should be conducted is often driven by regulatory
or standards requirements. If a company is able to set its own schedule, a risk-based
approach such as that described by DeWitt (2008) can be used.
A concern sometimes expressed about audits is that they create ‘‘smoking guns’’, i.e., they identify problems which, if not addressed, could create liability – particularly if
a serious incident occurs in which that deficiency was implicated. However, implicit in
the decision to conduct an audit is the commitment to address any problems that are
uncovered.
Although an audit may be triggered by an external event, such as a law suit or
a serious accident, most audits are internal and are conducted according to a pre-
established audit cycle. Some of these voluntary audits are carried out as if they mock
regulatory audits, i.e., the auditors behave as if they are regulators who are investigating an accident. Not only does this tactic help to identify problems related to regulatory
non-compliance, it may also reduce a company’s exposure to penalties should there
actually be an accident, because it demonstrates the company’s commitment to doing
a good job. In addition, mock regulatory audits familiarize the plant personnel with what
a real audit might be like should there ever actually be a major event.
Reasons for audits
Some of the reasons for conducting an audit are discussed below.
Accident follow-up The most serious type of audit is that which is carried out following an accident in which
someone was badly hurt or killed, and/or in which there was a major environmental
release. Although rare, such audits can be expected to be both serious and very
adversarial. The persons being audited must realize that the auditors (regulators and
opposing attorneys) are looking to find fault and to assign blame. Specifically, they are looking for evidence that will justify the prosecution of the facility’s management, and/
or that will provide the basis for large financial claims. This type of auditor has no
interest at all in helping the facility management improve its performance.
Regulatory/standards compliance The second most serious type of audit is that which is conducted by an agency such as
OSHA or the EPA to see if the pertinent laws and regulations are being followed. (Such
audits are sometimes triggered by a complaint by an employee or a member of the public.) Although there may not have been an actual accident prior to the audit, the
audit findings could result in serious financial penalties and even criminal action. Once
more, the auditor and the facility personnel are most definitely not on the same team.
This means that those being audited must be careful about revealing out-of-scope
problems.
Voluntary check Facility management may decide to conduct a regulatory-style, mock audit on their own initiative in order to identify any weak spots should a real audit ever occur. Such audits
are often conducted by companies that believe that they are in good shape but they
want to be sure that nothing has been overlooked. For example, the OSHA process
safety management regulation requires that operating procedures be validated annually.
It is possible that the procedures at a particular facility are of very high quality, but have
not been formally checked within the last year. An internal regulatory audit will high-
light problems of this nature.
Companies that conduct voluntary checks tend to fall into one of the following three categories:
1. We are the best. Some companies believe that they have an excellent opera-
tional integrity/PSM program but they want the auditor to double check this belief.
2. We want to improve a good program. Other companies believe that they have
a good program, but recognize that it can be improved. They want the audit to
provide them with guidance and suggestions as to how to go from good to best.
3. We have nothing. The third type of company is one who recognizes that their
program is severely deficient, and that fundamental changes are required. The audit establishes a baseline and a starting point.
Mock audits may help to reduce a company’s exposure to penalties should there
actually be an accident because it demonstrates the company’s commitment to doing
a good job.
Formal audits 697
Insurance and business security Insurance companies sometimes conduct audits – generally for one of two reasons. First,
they want to establish the correct premium payment. Second, the insurance company
wants to help their client company operate more safely, so that there is less chance that
payments will have to be made, and that, if they are made, that they of a smaller size.
Sometimes audits are carried out for business or commercial reasons. For example, if a facility is to be sold, the buyer may elect to conduct an operational integrity audit as
part of the due diligence process.
Audit standards
The first step in the formal audit process is to decide on the standard that is to be used as
the basis for the investigation. Typically, there are three types of standard: regulations,
industry codes and internal standards.
Regulations Management’s first priority must always be to make sure that they are in compliance
with the law. In some ways this type of audit is the easiest to carry out because the audit
questions are pre-defined. All that the auditor need do is take a regulatory statement or
sentence, and invert it to create the audit question. For example, the OSHA PSM stan-
dard to do with hazards analysis contains the following requirement:
(3) The process hazard analysis shall address: . (v) Facility siting;
The matching audit question becomes:
Did the process hazard analysis address facility siting?
Duguid (2004) has published a series of checklist questions that can be used for
a process plant safety audit.
Industry standards The auditor may be asked to check the facility’s performance against industry consensus
standards from organizations such as the American Petroleum Institute (API), the
National Fire Protection Association (NFPA), and the American Society for Mechanical
Engineers (ASME). Such standards, which are discussed in Chapter 5, represent an
industry-wide consensus as to what constitutes safe and effective performance. They may
not carry the full force of law, but they are authoritative, and are often cited in lawsuits.
Many companies have also committed to quality control standards, such as ISO 9000
and ISO 14000. Given that there is a good deal of overlap between such standards and an OIM program, an ISO audit may also contribute to an OIM audit.
Internal standards Many companies find that industry standards are not detailed or specific enough to
cover the particular circumstances to do with their particular operations and tech-
nology. For example, a company that manufactures a hazardous chemical that is used by
very few other organizations may choose to write its own standards for handling that
chemical. These internal standards can be used as the basis of an audit protocol.
Audit personnel
An auditor must always be independent of the facility being reviewed and must always
have complete freedom to observe and report on all aspects of the operation. His/her outside perspective not only ensures that there is no conflict of interest; it can also
provide alternative points of view that can be helpful to the people working within the
facility if he/she is asked to offer advice following the audit proper.
Regardless of their experience or background all auditors must share one attribute:
they must be of high integrity. There should be no doubt in anyone’s mind that the audit
is being conducted fairly, and against objective standards. There should be no suspicion
that the audit is ‘‘wired’’ to give a desired result, or to protect certain individuals or
departments. Most auditors are paid by the facility being audited. Hence, in principle, they can be
influenced by those who hired them. However, a good auditor knows that his/her
professional reputation rides on his/her objectivity, integrity and balance. Consequently,
the auditor will report the facts, as he/she sees them, regardless of external influences. If
toes get trodden on, then toes get trodden on. The auditor makes his/her living by
maintaining professional independence and integrity.
Outside auditors Outside auditors generally come from one of the following four groups:
- A regulatory agency;
- An outside company that specializes in process safety work; or
- A person from within the client company, but based in another department.
The use of outside companies should make the audit more objective. Outsiders are
not likely to be familiar with the processes being audited, nor are they involved with the
inter-personal relationships on the unit. For them, this project is only one of many, so
they will not be hesitant about issuing critical findings. A second benefit of using an
outsider is that he/she will bring a fresh perspective based on his/her knowledge of how
other companies and facilities operate. In particular, he/she may identify hazardous situations that employees at a facility have learned to live with, but that would not be
accepted as being safe elsewhere.
An outside auditor should not provide follow-up services to the audit. Such services
represent a conflict of interest, with the exception of reviewer-type comments, as dis-
cussed above.
Internal auditors Large companies sometimes appoint auditors from other departments or facilities within that company. These people act as if they are completely independent of the
facility that they are auditing. In practice, achieving this independence can be difficult
for three reasons.
First, the company as a whole may have some blind spots that are common to all of
their facilities. Every company has its own culture, and it may be that internal auditors
will not realize that such blind spots exist.
Formal audits 699
The second potential problem is that internal auditors must rise above their own
knowledge of company problems and internal personal relationships. Such knowledge
may not be explicit, and may even be unconscious, but it is still a factor, and may affect
the auditor’s judgment.
Finally, it has to be recognized that internal auditors and those whom they are auditing ultimately work for the same bosses. Their chains of command may not join
until high up in the organization, but they are joined. Consequently, there will always be
concerns – whether justified or not – that the auditor is not truly independent. He/she
knows that, sooner or later, it is possible that he/she may be working directly with or for
the people being audited. In point of fact, most auditors would not actually let such
matters affect their judgment. But auditing is not just about facts – it is also about the
perception of those facts, as summed up in the Chinese proverb, ‘‘Do not tie your shoes
in a melon patch; do not adjust your hat under a plum tree’’. So with auditing, there needs to be a perception of detachment and objectivity.
The auditor should not be someone who works at the facility being audited. In
addition to the issues discussed above, people working within a facility will not see it
with an outsider’s eyes. They may lack ‘helicoptic vision’, i.e., the ability to rise above
the minutiae of a problem to see the surrounding countryside. In other words, they can’t
see the forest for the trees.
When a company is conducting a review rather than an audit, the appearance of
independence is not nearly so important. Nevertheless, it still makes sense to use an outsider to lead the review because he/she will bring fresh points of view and fresh
insights to the way the facility is operated, which is one of the most important reasons
for conducting the review in the first place.
Team composition A short audit of a simple unit may be carried out by a single individual who is knowledgeable in the process. If the scope of the audit is larger, specialists will be
needed to make sure that the right questions are being asked and answered (rather like
a process hazards analysis). The team could include personnel from the following
areas:
Auditor attributes
Audits are stressful for those being audited. No matter how good a plant’s performance
may be, there is every chance that the auditor is going to find something that indicates a problem; and most problems and deficiencies are eventually traced back to specific
individuals. Therefore, the personal attributes and demeanor of the auditor are
extremely important. Some of these attributes include the following:
- Interview skills;
- Technical knowledge;
- Demeanor.
Interview skills A critically important skill for an auditor is interviewing. In particular, he/she needs
to be someone who can ask open-ended questions and is a good listener. As discussed
in Chapter 12 (Incident Investigation) the auditor should listen to what people
actually say, focus on facts not opinions, and establish trust with the people they are
talking to.
The auditor will have to work with a wide variety of people. He/she will be inter-
viewing a full gamut of personalities and job descriptions, ranging from the facility
manager to temporary workers who have no knowledge of the plant culture. This means that the auditor will have to be empathetic, i.e., he/she needs to understand where
people are coming from, and what they can be expected to know. (Empathy is different
from sympathy in that the auditor may relate to a person’s concern, but still retains his/
her detachment and objectivity.)
In addition to listening to what people say, the auditor should also be able to read
body language. Some workers may have trouble verbalizing what they want to convey –
but their body language may indicate that the auditor should pursue a certain line of
inquiry. The auditor should keep good notes that document the results of interviews, docu-
ment reviews and equipment testing. These notes will be essential for communicating
findings.
Technical knowledge The auditor must possess good technical knowledge of the process industries. It is
unlikely that the auditor will be an expert in the technology of the process being investigated. However, he/she should possess a good understanding as to how process
plants work, and what items to look for. The auditor also needs to be familiar with
regulatory requirements in the industry, and should have a good knowledge of the
pertinent codes and standards.
A catch with using experienced people for formal audits is that they may be too
sympathetic to the persons that they are auditing. Hence, they will want to try to help,
not to evaluate; thus degrading their objectivity.
Writing skills The final product of any audit is a written report. Therefore, the auditor should possess
good writing skills. He/she must be able to quickly prepare a report that is complete,
readable, accurate and fluent. It is good to deliver the report quickly while everyone’s
Formal audits 701
mind is on the audit and has not moved on to the next set of activities. General guidance
to do with report writing is provided in Chapter 16.
Demeanor Throughout the audit, the auditor must be pleasant and empathetic and conduct himself in a professional manner. The auditor should not make any negative comments during
the audit process, but he should always express appreciation for the effort that the client
is making to provide information and assistance.
Auditors need to be circumspect d they should not make off-the-cuff comments that
could be misinterpreted and exaggerated by those listening, nor should they offer
comments on matters that are not related to the audit itself. In other words, throughout
the audit, the auditor must be friendly and empathetic without losing sight of the fact that
he/she is an outsider who is not involved with local organizational or personal issues. Occasionally the auditor will have to work with one or more persons who are openly
antagonistic, uncooperative or even hostile to the audit process, and to the auditor
himself. In such situations the auditor must not give up on digging out the necessary
facts, while always maintaining a professional demeanor.
The auditor must be careful not to personalize his/her work. The identification of
a deficiency does not symbolize success, nor is it a personal triumph. The auditor’s job is
to report facts as dispassionately as possible.
The host company
It is often necessary to brief those people in the company that is being audited on how
to work with the auditors. Items to be considered include:
- How to answer all questions fully and honestly, without volunteering information
that does not pertain to the question being asked.
- The use of gifts. Inspectors from some government agencies, for example, cannot accept even a cup of coffee.
- How to respond when deficiencies or problems are uncovered.
- The purpose of the opening meetings.
- Whether to allow the auditor independent access to people and records, or
whether the auditor should always be accompanied by a staff counterpart,
who will serve as a watch-dog.
- When to refuse to answer questions on grounds of not having sufficient authority or knowledge to make a trustworthy reply.
- Knowledge of which records are open for inspection.
- Whether to take matching information. For example, if the auditor takes pictures,
his/her escort must know whether to take a picture of their own at the same time
so that there is independent verification of the auditor’s findings.
Those being audited should always keep in mind the importance of the impressions
that they make. An auditor will be impressed not only by a high quality process safety or
operational integrity program, but also by one that looks good, i.e., one where the
documentation is neat and tidy, and it is easy for the auditor to find his/her way around
the system.
First impressions The facility management needs also to recognize that auditors are human; as such, they
will always be impressed by a program that looks good. In particular, management
should never forget that ‘‘you don’t get a second chance to make a first impression’’.
For example, if a facility has written first-class operating procedures, it makes sense to
publish them in high quality binders, with lots of color pictures included with the text.
The auditor who reads these procedures is likely to be impressed. This does not mean
that appearance is everything – the procedures themselves must also be complete,
accurate and up-to-date. Nevertheless, their quality will appear much greater if they also look good.
Employees Just as the auditor should conduct himself/herself in a professional manner, so all
employees at the facility being audited should behave calmly and without emotion –
regardless of how they really feel. This is particularly true if the audit is basically hostile
in tone – presumably following a serious accident. In particular, employees should
understand that it is their responsibility to answer questions truthfully, but to say no more than the question requires. Especially they must learn not to volunteer informa-
tion. Such information could be used against them or their company.
On occasion, the persons being audited may not know the answer to the questions
posed by the auditor. They must state that they do not know an answer, or they do not
have the required information. They should then assist the auditor with finding the
information from someone else in the facility.
Planning the audit
Audits are generally organized around the steps shown in Table 14.2.
Table 14.2 Audit Steps
2. Establish the audit standards to be followed;
3. Define the scope and budget;
4. Conduct the audit;
5. Issue a report;
7. Follow-up; and
Formal audits 703
Like any other management activity, an audit should be properly planned. In
particular, as many as possible of the administrative details and information acquisition
steps should be carried out before the audit team actually starts work. The plan should
clearly define the objectives of the audit, the geographical and organizational scope of
work, and the anticipated amount of time that will be needed.
Goals Before the audit starts, the facility management must decide what it intends to gain from
the audit, in addition to simply verifying that they are in conformance with a regulatory
standard. For example, the auditors can be asked to measure compliance with additional
standards such as ISO 9000. Also, the audit team needs to know whether the present
audit is meant to evaluate compliance with the recommendations and findings of the
previous audits. Topics that can be raised while setting goals include the following:
- The business objectives of the facility;
- Safety or environmental targets;
- Management philosophy and organization;
- PSM objectives; and
- Regulations and standards.
Determine the audit standard The next step in the audit process is to create a standard or protocol. With regard to risk
management and process safety management, the standard is generally provided by an
external organization such as a regulator or corporate office.
Scope The physical or geographical scope of the audit must be clearly defined. Figure 14.1 is an example to show how this can be done. The shaded area is to be audited; the remaining
area is excluded – including the process links between Areas 100/200 and the remainder
of the facility.
It is important to decide to what extent utility systems are to be included in the audit,
because they provide physical connections between units, utilities run everywhere. For
example, systems such as steam, cooling water and instrument air are to be found in all
parts of a facility. Therefore, if the auditor is analyzing just one section ofa facility, he/she has
to know exactly which utilities are included, and where the boundaries are to be drawn. The problems associated with connectivity and utilities have provided the basis for
much legal discussion with regard to process safety. After all, all plants are connected to
the ‘‘outside world’’ in many ways, often through utility systems such as electric power
and water sewers. The existence of these connections provides connectivity (in the
legal sense) between the process and outside facilities. These issues need to be resolved
before the audit starts.
The audit team also needs to establish whether stand-alone items, such as skid-
mounted units that are provided complete and ready-to-go by outside companies, are to be included in the scope.
Area 100 Area 200
Budget The final part of the planning process is to develop a budget. Since process safety is an on-
going program, there really should be no need for special efforts before the audit starts.
However, there will be some costs associated with the audit work. For example, one
employee may be designated as the point of contact for providing written information to the auditors. If the audit is known about ahead of time, some of the facility’s documen-
tation may be reprinted and re-organized, but such activities should be minimal. If the
auditor is hired from outside the facility there will be a direct out-of-pocket cost. Qualified
auditors usually charge a high hourly rate because their skills are not widely found.
In addition to funding, quality personnel are needed. On the facility side, there will be
some pre-audit preparation. Typically, some of the necessary documentation will be
collected and organized so that the auditor’s time is not wasted. It may also be necessary
to print fresh copies of pertinent documents. However, the major use of personnel time will be during the audit itself, when key personnel are interviewed and records are
researched.
The biggest cost of most audits is the funding needed to carry out the recommen-
dations that address the auditor’s findings. However, as has already been noted, if
a company commits to being audited, they have also committed to the investment
required in the follow-up.
Schedule Although most process safety audits are quite short (typically just a few days) it is still
important to prepare a formal schedule. Not only does a schedule minimize problems
such as two auditors wishing to speak to the same person at the same time, it also is
a courtesy to the host company. Most people are very busy, and they do not want to
waste time waiting for an auditor to show up.
Generally, the order in which the elements of the risk management programs are
audited is not important. Logistics often force a particular schedule to be implemented.
For example, if the plant maintenance engineer is only available on a particular day, then that will be the time to review mechanical integrity with him. However, the order in
Formal audits 705
which items are discussed may be directed by the results found to that point. For
example, if a problem to do with the emergency procedures is found, the auditor may
suspect that the same problem will occur with the operating procedures. Hence, he may
choose to review both of these items at the same time.
Leaving these considerations aside, the auditor should start with those items that are likely to have general application throughout the entire facility. That way, if he/she finds
a problem, his/her subsequent investigation will serve to confirm or deny that first
impression. For most plants, therefore, the following three items provide a starting point:
- Employee Participation because of its central importance to process safety.
- P&IDs because they provide the informational base.
- Management of Change because of its criticality in preventing uncontrolled change.
One-point contact It is crucial that both the audit team and the facility each have a one-point contact. All
communications such as requests for information and schedule adjustments should go
through these persons.
Pre-audit activities Before the audit starts, the host company will probably spend some time preparing for
their visitors (unless the audit is unannounced). This period of time can be used to make
corrections to items that obviously require attention.
Audit forms Audit forms will usually contain four major sections:
1. A question that refers back to the appropriate standard.
2. A check box that provides space for the answer to the above question.
3. Space for recording where the information came from.
4. Space for discussion and comments.
Usually, each question has one of four possible answers:
1. Satisfactory d meets requirements
2. Unsatisfactory d does not meet requirements
3. Not audited
4. Not applicable
Table 14.3 provides an example of an audit protocol design (the question addresses
OSHA’s Process Safety Information element).
- The first square in the Protocol contains the number 3.9. This is the number of this
particular question. Below it is the question itself; ‘‘Are inadvertent mixing
scenarios anticipated?’’ This question matches the requirement that the employer
must compile information to do with the ‘‘Hazardous effects of inadvertent mixing of different materials that could foreseeably occur’’.
Table 14.3 Audit Protocol Design
3.9 Are inadvertent mixing scenario anticipated?
Y
N
N/A
Facility
Area
Chemicals
- The next two squares contain a Y/N representing a Yes or No response to the
question. There is no ‘‘Partial’’ response; a ‘‘Partial Yes’’ is recorded as ‘‘No’’.
There is, however, an N/A space that stands for ‘‘Not Applicable’’. Not all plants
have to meet all the requirements of the standard, so space must be provided for
this.
- The next two rows (Facility and Area) describe to which areas the question is
being applied.
- The Chemicals row lists those chemicals that are present that could cause an acci-
dent if accidentally mixed.
- The next row provides space for the names and titles of the people who were interviewed in order to obtain the information necessary to answer the audit
question. Similarly, there is space for the names of the documents that were
reviewed as part of this audit.
- Finally, there is a Notes section. If more space is needed, observations can be recorded separately, with the notes linked to the question number.
Formal audits 707
Conducting the audit
An audit generally consists of a combination of interviews and review of documentation.
Ideally the one verifies the other. For example, the auditor may ask an operator how he performs a certain task (the interview). Having been told, he will then ask for a written
procedure that confirms this (documentation). Or the auditor may review a Material
Safety Data Sheet (MSDS), and then ask the operator to verbally confirm that he
understands the major safety issues to do with the chemical in question.
The auditor will have to be selective regarding the documents that he/she reads and
the people who are chosen for interviews. There simply is not enough time to read
and discuss everything. An experienced auditor will sense where the soft spots are
and, based on the information received to that point, will use his/her experience to choose the follow-up questions. For example, if the operating procedures on the facility
being audited are much shorter than the operating procedures on other similar facilities,
he/she may choose to investigate this element with increased scrutiny. The shortness of
the procedures may indicate that facility has a deficiency in this area (or it may mean that
they were very well written).
Formal audits must be objective and all findings must be based on observed, verifiable
facts. Ideally, two auditors would reach the same conclusions when asked to analyze the
same situation with regard to the same baseline standard. If the auditor does provide a opinion, it should be very clear that he/she is doing so.
Auditor preparation The auditor needs a good general knowledge of processes in general in order to be able
to conduct an effective audit. However, the auditor cannot initially have a detailed
knowledge of the facility or of the process that is to be audited.
It is suggested that, at this stage in the audit process, the auditor should learn the
details of the process and the facility that he/she is about to audit. This includes learning
the client’s organizational structure, the equipment being used, plant capacity, age of the facility and of the equipment, and the means used to bring materials into and out
of the facility. This work must be done before the audit starts, otherwise the auditor will
waste a good deal of everybody’s time while he/she gets up to speed.
Kick-off meeting The audit will start with a kick-off meeting. The audit team and representatives from the
client company will meet to discuss the way in which the audit is to be carried out.
Basically they will discuss four items:
1. The objectives of the audit d why it is being performed.
2. The mechanics of the audit, including issues such as schedules, budgets and facil-
ities to be covered.
3. The scope of the audit: including regulations to be covered.
4. The depth of the audit questions on technical issues.
The kick-off meeting is not a one-time affair. Every time a new person from the facility
is brought into the audit, he/she should have his/her own kick-off meeting, that
provides him/her with an explanation as to what is going on, and what is expected of
them. In particular, it must be made clear to those people who have not been through an
audit before that the aim is not to find fault with individuals, but to identify limitations in
the management systems.
During the kick-off meeting, background documentation, including items such as
those listed below, can be made available:
1. Permits;
4. MSDS and other safety information;
5. An organization chart;
6. A plot plan, including information on population concentrations such as schools, hospitals and prisons;
7. Process description; and
8. Previous audits and inspections (including the action items that they generated).
Either at, or just before the kick-off meeting, the schedule for the audit should be
developed. The creation of a schedule is particularly important if the audit is being
carried out by a team.
Plant tour The inspection will usually start with a plant tour, during which the auditor will obtain
a general overview of the facility and its operations. During this tour, a management representative should accompany the inspector – particularly if the auditor is from
a government agency or is representing a plaintiff. The auditor should inspect areas
which may normally be overlooked, such as platforms at the top of large columns or
behind the control panels.
collecting and reading documents, interviewing operators and maintenance personnel and conducting a plant tour. Wherever possible, auditors should try to cross-check
information with information from a different source. For example, the auditor can
check an operating procedure by asking an operator to demonstrate how a particular
task is carried out in the field. Or he/she can check the Management of Change system
by ensuring that a field change that is currently taking place was properly analyzed, and
that it was marked up on the P&IDs.
Interviews The interview is at the heart of most audits (interview skills were discussed earlier in this
chapter). The auditor systematically works through a series of questions with one or
more of the people who work at the site. Typically, the process follows these steps:
1. The auditor asks the interviewee to describe his/her work. This gives the auditor a sense of the boundaries for this particular interview.
Formal audits 709
2. The auditor then works through a questionnaire that corresponds to the pertinent
standard. The OSHA standard with respect to emergency response, for example,
requires that ‘‘the emergency action plan shall include procedures for handling
small releases’’. The auditor will ask to see the emergency action plan, and will
ensure that it contains provision for handling small releases.
3. The auditor will then pursue the question in more detail, often using the other
elements of the standard for guidance. So, using the above example of emergency
response to small releases, the auditor will ask the interviewee for copies of the
procedures to handle this scenario, evidence that the operators have been trained in it, documentation that contract workers know what to do in the event of a small
release, and so on.
4. The auditor will then move on to the next question. He/she is always watching
out for systemic problems. If one particular procedure is not in place, that may be an oversight. If, however, it is found that there is a general lack of emergency
procedures, this indicates a more fundamental management problem. The
auditor can also use the first answers as guidance for where to focus later on
in the audit.
During the interview the auditor must take notes, otherwise he/she will not be able
to verify his/her findings. The note-taking should be as discreet as possible, so that the
interviewee is not distracted or worried.
In principle, voice recorders are an excellent way of capturing information during
interviews. No matter how skillful the auditor may be at capturing statements in his/her
notes, it is inevitable that he/she will miss some of the statements that are made. If the
interview has been recorded, the auditor has an opportunity to go back and make sure that he/she captured all the information provided to him/her. In practice, however,
recorders are rarely used, because the person who is speaking may fear that any off-the-
cuff comments could come back to haunt him/her. Consequently, the interviewees
become very cautious, to the point where they simply do not communicate the infor-
mation that the auditor needs to do his job properly.
The auditor should use a mix of open and closed questions. An open question such as
‘‘How are contract workers trained?’’ invites a discursive response. A closed question
such as ‘‘Are the contract workers trained?’’ invites a Yes/No response. Leading questions such as ‘‘Have you started training contract workers, yet?’’ should be avoided
because of the bias that they introduce.
One problem that sometimes occurs with regards to interviews is that the inter-
viewee’s manager or supervisor may wish to be present during the conversation. It is up
to the auditor to decide if this is acceptable (with a regulatory audit, legal requirements
may come into play). The presence of a manager may inhibit the interviewee, and
prevent him/her from describing the situation as it is, rather than as it should be.
Moreover, the manager may attempt to direct the audit in ways that make him/her look favorable. This is particularly the case with issues to do with communication and
training because of a possible gap between theory and reality. A manager may point out
that written policies and programs are available for these topics, but he/she may be
reluctant to have an auditor speak to an operator, because the operator is completely
unaware of these written policies.
On-site inspection The auditor should verify information by field inspection. For example, an operator may
have a procedure for carrying out a task, but, when demonstrating what he/she actually
does when working with the plant equipment, may demonstrate that his/her actions are
not in conformance with the procedure.
Close-out meeting At the conclusion of the audit, the audit team should have a close-out meeting with the
host company. This meeting will provide an opportunity for the auditors to provide an
overview of their results, and to highlight any serious issues that may have been iden-
tified and that require immediate attention. The response of the facility personnel to
these preliminary findings can be incorporated into the final audit report. If agreeable to
both parties, it is possible to have debriefing meetings as the audit progresses
(government regulators would probably not agree to this).
Report
Once the audit has been completed a report must be issued. An audit report should
stress the reasons for conducting the audit, and should make explicit the standards
against which the audit is being conducted. Discussion as to the organization and
content of a representative professional report is given in Chapter 16. Specific
comments to do with audit reporting are provided below. As with Process Hazards Analyses (Chapter 3), the auditor should prepare a draft
report as quickly as possible. No matter how effective the auditor’s note-taking may have
been there will always be some observations and facts that were not properly written
down. These should be captured before the auditor forgets them. It also gives the client
an early opportunity to correct any factual errors in the report.
Where feasible, the report should be presented in a meeting, particularly if it
contains difficult or sensitive findings. The auditor should expect to be questioned
and challenged as to his/her conclusions, and it is best if he/she can do this face to face.
How the draft report is handled at this point will depend on the relationship between
the auditor and the client. If the audit is truly formal, then the facility personnel will not
be provided with a chance to review it (this is particularly true if the audit is by
a regulatory agency, or is organized by opposition attorneys). However, if the audit is
internal there should be no problem in having the facility personnel look over the
findings before the final report is issued. They cannot change the overall tone of the
report, nor do they have the authority to change any of the statements in it. Nonetheless, they may be able to point out factual errors, or identify significant typographical
problems that could lead to a misunderstanding.
Sometimes a client will wish to add information concerning action that was taken
immediately following the audit, particularly if the auditor had identified a serious
problem that was corrected right away. This type of information should not be
included in the audit report itself, which is based on observations at a particular
time.
Formal audits 711
It is important that the auditor has someone within his/her own company to review
the report before the client sees it. The purpose of this review is to catch any obvious,
and potentially embarrassing, errors. Items to watch for include:
- Inconsistency between reported facts;
- Completeness, particularly with respect to the standard being used.
At all times, the auditor must be careful to distinguish between what was
observed and what he/she deduced, and must avoid the trap of slipping into
unconfirmed generalities. For example, he/she may have asked for copies of the
inspection programs for three different pumps. On finding that there is no inspec-
tion program for these pumps, he could write, ‘‘There is no pump inspection
program’’. Actually, the finding should read, ‘‘Inspection programs for pumps . were not available’’. He/she may then go on to say, ‘‘Based on the above finding,
there is reason to believe that the pump inspection program is deficient’’. Similarly, if he/she notes on two occasions that the facility was carrying out temporary
operations without having written procedures for them then the report should state,
‘‘Observed that two temporary operations were being carried out without written
procedures’’. The report should not state, ‘‘Facility does not have a policy for writing
temporary procedures, as evidenced the following two incidents’’. Even if someone
tells the auditor that a policy for writing temporary operating procedures does not
exist, the auditor should make it clear that he/she is reporting that statement. The
only conclusion that the auditor can draw from that statement is that that person is not aware of the existence of such a policy, not that such a policy does not exist
at all.
The above comments do not mean that there is no place for general conclusions and
deductions, particularly in reviews as distinct from audits. However, such conclusions
should be identified as such. In the above example, the auditor might write, ‘‘There
appears to be no policy for writing operating procedures for temporary operations. This
conclusion is based on the following observations..’’
Letter of certification The audit should be certified by the auditor. A representative Letter of Certification is
provided in Table 14.4 for use in process safety management work.
Audit verification This section provides a clear description of how the audit met the requirements of the
standard or regulation against which the audit was being conducted. Table 14.5
provides an example. It lists the paragraphs associated with the auditing standard in the
left-hand column (in this case taken from the OSHA regulation). The statements in the
right-hand column show what was done to in this particular audit in order to comply
with those requirements.
Table 14.4 Representative Letter of Certification
The attached audits meet the requirements of the OSHA process safety management standard,
29 CFR 1910.119.
- The audits were conducted within the three-year program required (paragraph o.1).
- One of the auditors (the process safety manager at the facility) was knowledgeable in the
process (paragraph o.5).
- One of the auditors (from the auditing company) was knowledgeable in audit techniques (paragraph
o.5).
- Sample sizes were big enough (paragraph o.8).
Findings
All findings must have enough detail to implement and verify closure. Findings that are vague or that cannot ever be addressed should not be issued. As with hazards analysis,
a clear distinction between findings and recommendations is required. An audit team
issues findings; it does not make recommendations.
Table 14.5 Sample Audit Verification
Requirement Action taken
section at least every three years to verify that the
procedures and practices developed under the
standard are adequate and are being followed.
This report includes a letter of certification.
It also shows that the audit was conducted
within the three-year schedule required by the
regulation.
by at least one person knowledgeable in the
process.
resume is included. It shows that he/she is
knowledgeable in the process.
(3) A report of the findings of the audit shall be
developed.
(4) The employer shall promptly determine
and document an appropriate response to each
of the findings of the compliance audit, and
document that deficiencies have been corrected.
A Follow-Up report will be prepared based on
the findings of this audit. This report will
develop recommendations that address each of
the findings. Also, it will include a process for
making sure that all recommendations are
executed and closed out in a timely manner.
(5) Employers shall retain the two most recent
compliance audit reports.
The audit reports are retained as part of the overall
process safety documentation.
Formal audits 713
Follow-up
The audit does not stop with the release of the auditor’s report. Management is required
to address the audit’s findings – possibly with the assistance of the auditor – to improve management systems. The purpose of an audit is to help management improve
performance, not to find fault, or to establish blame (although it has to be recognized
that one way of improving performance may be to change some of the managers).
Facility management must treat the audit as being part of the continuous improvement
program rather than a law enforcement activity.
The auditor may be asked by the client to use the findings to identify manage-
ment problems that lie behind the issues that were reported on. To do this, the
approach used in Chapter 12 for Incident Investigation can be followed. That is, the elements of process safety are used to structure the analysis of the management
systems.
As part of the follow-up, the company that was audited should tell the audit team how
they felt about the exercise and the value of the findings that came out of it.
Unannounced audits
It is important always to be ready for an unannouced audit. There is no telling when an accident might happen, and if the accident is bad enough, there will be audits from
regulators and attorneys representing various plaintiffs. The fact that a facility is in the
middle of developing its risk management program is not a valid excuse; an accident can
happen at any time, therefore an audit can take place any time.
The practical implications of this understanding are twofold. First, it means that all
information to do with risk management must be properly organized and indexed so
that it can be quickly located. Doing so will create a good impression with the auditor,
and it will reduce the possibility of not being able to find the information that was asked for.
A second implication of being ready to meet an audit at all times is that it strongly
encourages a top-down approach to developing a risk management program (Chapter 16).
Initially, there will be very little detail in the program. But, as it develops, more detail will be
added. The key is that, at all times, the program will be complete, and it will not be
a collection of miscellaneous documents that may or may not be connected to one another
in a coherent manner.
Additional advantages to having a top-down approach include the following:
1. If the plant is audited, there is always a complete, coherent, integrated and
holistic program to show the auditor. He/she will probably be impressed with
the organization and control of all the risk management activities that this exemplifies.
2. It helps to sustain the employee morale and enthusiasm. Risk management
programs are hard work and can become debilitating for those involved in
it; the completion of stages on a regular basis will help to create a sense of progress.
3. It will be relatively easy to identify those areas that need the greatest attention.
LAGGING AND LEADING INDICATORS When evaluating any management system it is important to quantify the results where
possible. Quantification allows management to measure progress over time, and it also
allows different facilities and companies to be compared with one another. In practice, the
quantification of risk management programs is difficult, particularly with respect to the
more intangible elements, such as employee participation and management of change.
In order to establish reliable quantification measures, a consistent set of terms and
reporting standards is required. In the area of occupational safety, considerable stan-
dardization has already been achieved through the use of measures such as the number of first aid cases or recordable injuries. Although different organizations will apply these
terms slightly differently from one another, there is sufficient consensus to allow their
use across broad swathes of industry. For process safety, it is much more difficult to
come up with comparable yardsticks. Hence comparisons between different facilities
may lack validity and credible trend lines are difficult to develop.
Lagging indicators
Lagging (sometimes called trailing) indicators are widely used – particularly for occu-
pational safety and equipment reliability – to measure performance. These indicators
include well-established parameters such as lost time accidents, first aid cases and
recordable injuries. Figure 14.2 illustrates how the indicators are tracked over time.
Lagging indicators are widely used because, assuming that there are enough events to
ensure statistical significance, they allow management to establish baselines, measure
trends and to compare results with other facilities and companies.
Some companies define various key performance indicators (KPIs) as lagging indicators to be watched particularly closely. One oil company, for example, has set
the following KPIs for itself (some are monthly, others quarterly and the remainder
annual).
- Fatalities;
- Recordable illnesses;
- Spills from primary containment (even if secondary containment was effective);
- Spills affecting the environment (failure of all containment barriers);
- Volume of oil spilled that is not recovered;
- Greenhouse gas emission equivalents;
- Total SOx and NOx emissions;
- Total discharges to water; and
- Total hazardous waste energy use.
Companies in the United States pay particular attention to the OSHA recordable rate
that facilities report. An OSHA recordable injury is an occupational injury or illness that
meets one of the following criteria:
- Death;
- Medical treatment beyond first aid.
It is calculated for the previous three years and is defined as:
Number of recordable cases 200; 000
Total hours worked
Number of lost workday cases 200; 000
Total hours worked
A lost workday – equivalent to a lost time injury – is one where an individual misses more than one day of work due to an injury sustained while at work, and is another
widely used criterion for measuring occupational safety.
Lagging indicators by themselves do not provide much explicit guidance to manage-
ment as to what needs to be done to keep improving safety. The events themselves have
to be analyzed. Also, lagging indicators tend to react quite slowly to system changes.
It is difficult to identify effective lagging indicators for use with process safety. The
most obvious difficulty is that major process safety incidents do not occur frequently
enough to develop a statistically significant trend such as that shown in Figure 14.2. If
many facilities and companies pool their data it may be possible that some trending
results can be developed. However, such results are always open to doubt, not least
because different organizations define terms differently. For example, the Baker (2007)
report provides a list of events that fall under the term ‘‘fire’’. That list includes ‘‘a fault in a motor control centre’’. It is questionable as to how many organizations would call such
an event a ‘‘fire’’ unless it resulted in flames and/or smoke.
An additional difficulty is that many process safety events – particularly those that are
near misses – may simply not be recognized for what they are. For example, an operator
and a mechanic may fix a leaking pump seal, not realizing how close they were to having
a major accident.
Leading indicators
A second approach to quantifying process safety is to examine leading indicators.
Leading indicators are not to do with incidents per se; they are merely events that
provide management with information as to how the future may look. Examples of
leading indicators are shown below:
- Number of field visits and inspections;
- Number of safety audits;
- Percentage of incidents investigated;
- Number of positive rewards and recognition given;
- Safety communications;
- Number of safe behaviors observed.
For each of the topics shown above (and others like them) an increase in the number
of activities will, it is assumed, lead to an increase in process safety. Unfortunately it is
difficult to quantify the effect of activity-based leading indicators because there is no
way of knowing how much effect each activity has on overall results. For example,
a manager cannot know what impact, say, doubling the number of field visits will have
compared with doubling the number of positive awards ceremonies.
It is also vital to distinguish between activity and quality when analyzing process
safety data (Hopkins, 2007). For example, managers may be encouraged to closeout findings from hazard analyses and incident investigations more quickly than they did in
the past. If they do so then this leading indicator will show that the management of
process safety is improving. However, if those managers achieve the improvement by
not implementing the findings as thoroughly as before, the net effect may be a reduction
in performance. They score an ‘A’ for effort, but a ‘B’ for results.
Reviews and expert assessments 717
Limitations
Although leading and lagging indicators provide useful guidance with respect to safety
performance, they do have limitations, as discussed below.
Quality of reporting The effectiveness of both lagging and leading indicators depends heavily on the quality
of incident reporting. For lagging indicators this is not be a major problem here. As
already noted, indices such as OSHA recordables are widely understood and are quite
consistent across industries. Nevertheless, anyone who has worked in a plant knows
that many incidents – particularly the minor ones – are often not reported. Some reasons
for not reporting minor events include:
- Fear of looking bad, with the possibility of a reprimand or even being let go;
- Desire to ‘‘tough it out’’ or to appear manly;
- Not realizing how serious the injury might be (for example, unknown to the
person affected, a small scratch may have allowed toxic chemicals into his/her-
blood stream); and
- Desire to keep the numbers ‘‘looking good’’.
With regard to leading indicators, the quality of the reported data is likely to be worse
than it is for lagging results because it relies on the reporting of unsafe conditions and
near misses – not on actual events. The value of the reported results is likely to be very patchy and inconsistent. Some people are very diligent about reporting such events;
others are not. Therefore, it is difficult to establish a consistent estimating system –
particularly between different companies.
Management elements The reporting of leading and lagging indicators provides no guidance as to which of the
elements of process safety (Table 1.2) contributed to the system failure. In Chapter 12, it
was noted that different incident investigators can come up with widely disparate perceptions as to the root causes of events. Similarly with leading and lagging indicators –
different people will have different perceptions as to the causes of events.
REVIEWS AND EXPERT ASSESSMENTS The discussion to do with auditing stressed the formal nature of the audit process and
the need for objectivity. The auditor compares actual performance against a clearly
defined, agreed upon standard. He/she then provides a ‘‘Yes/No’’ result for the item under consideration. If the auditor moves beyond this restricted role he/she is acting as
a consultant, not as an auditor.
In practice, most clients want both a mock formal audit (in order to make sure that
they will ‘‘survive’’ a real audit from an agency) and an expert assessment or review, in
which the auditor/reviewer provides a more subjective analysis of the client’s process
safety systems. During an assessment the reviewer typically works as part of a team with
the facility personnel. Even though he/she is looking for problems, he/she is not trying
to find fault; any problems are identified as ‘‘recommendations’’ or ‘‘action items’’ d not
‘‘findings’’. There is no threat of penalties or retributive action based on his/her findings.
This type of review is not adversarial; the reviewer and the plant personnel are on the
same team. Moreover, the recommendations associated with a review can be considered against the reality of budgets, schedules, personnel availability and turnaround sched-
ules. Finally, a reviewer will often work with the facility being audited on developing
solutions to the problems that have been identified.
An analogy can be drawn between financial audits and economic analysis. A financial
audit checks that a company is conforming to the goals and standards set by external
agencies such as the tax authorities and the company’s own finance department.
Economic analysis, which corresponds to measurement and metrics, seeks to under-
stand issues such as the best pricing level for the company’s product, labor rates and how to finance the enterprise.
The distinction between audits and reviews was made clear at one facility at which
the audit/review team delivered two physical reports. The first was a mock OSHA audit.
When it came to the client’s operator training program, the audit noted that all regu-
latory requirements were being met fully – a training program was in place and was
being implemented. However, an assessment provided by the same audit/review team
stated that there were so many problems with the operator training program that the
best strategy may be to start a brand new program. An audit always results in a formal report, which is not necessarily the case with
regard to reviews. Moreover, the persons carrying out the review can come from the
same organization as the facility being audited; there is no concern about conflict of
interest. A reviewer will often work with the facility being audited to help develop
solutions to the problems that have been identified. He/she may be able to bring
experience and knowledge to assist the facility personnel in finding cost-effective
solutions to the problems identified. The auditor may also make recommendations
regarding the company’s management structure. He/she may suggest that a root cause of many of the problem areas was that the organization was not set up correctly, and that
alternative organizations may be more effective.
Review issues
- The effectiveness of management systems;
- Workforce involvement;
- Lessons learned from other facilities.
Management systems effectiveness Operational integrity management is – at its root – a management process. It provides a framework whereby managers can develop, implement and run systems that
Reviews and expert assessments 719
encourage all workers to operate safely. This being the case, the reviewer should eval-
uate the management systems by making sure that policies are in place, that the policies
make sense, and that they are properly understood and followed by all workers.
The reviewer should check for the following:
- Policies are in place;
- Those policies are appropriate and relevant;
- The policies are properly understood and followed by all workers; and
- Ensure that individuals within the host organization have the right levels of respon-
sibility, authority and accountability.
Workforce involvement The topic of workforce involvement is difficult to measure and audit because it is
concerned with the spirit rather than the letter of the program. Although it is quite
simple to have employees fill out forms and questionnaires, it is much more difficult to
determine if they are truly participating. Some questions that might help guide an audit in this area are listed below:
1. Is there an overall policy or mission statement that is perceived as being real, and
that is not just some form of management fad?
2. Does the process safety program have clearly defined target and objectives –
including dates when these will be met – and are the employees familiar with
these targets and dates?
3. Is upper management perceived as being committed to these objectives?
4. How well is the overall strategy translated into detailed plans?
5. Do the employees understand the program and the detailed plans?
6. Are they committed to it?
Real world usefulness The reviewer will check the process safety and operational integrity systems to see if
they are really adding value. For example, he/she may find out that the operating
procedures are almost never used – even in situations where they should be. He/she
should dig into this finding and try to learn why the procedures are not used.
‘‘Learned to live with it’’ problems Most people and facilities find that they eventually get used to certain problems, and,
rather than trying to fix them, they learn to live with them or they have developed work-
arounds. A good reviewer will identify these problems and point out that they need
fixing. Many housekeeping issues fall into the ‘‘learned to live with it’’ category.
Lessons learned The reviewer can examine events in other companies, and determine what lessons can be learned, as discussed in Chapter 12.
Reviewer attributes
The personality and experience of the reviewer is of paramount importance to the
quality of the review. To be effective in his/her analyses the reviewer has to have extensive plant experience. In addition to the personal attributes discussed under the
role of auditor, the reviewer should possess considerable experience of process facilities
similar to the one being audited. He/she should also have the strength of personality to
be able to communicate opinions and suggestions to all levels of management within the
host company. Ideally, the reviewer will also be able to generate creative solutions to
previously intractable problems.
MANAGEMENT ELEMENTS ASSESSMENT The previous sections of this chapter have looked at formal audits, lagging indicators,
leading indicators and expert reviews. An additional approach to quantifying risk management programs is through an assessment of management elements. This
approach addresses some of the limitations to do with lagging and leading indicators
discussed above.
Managers generally ask four questions regarding their risk management programs:
1. What is our current status?
2. Where are we most vulnerable?
3. How are we progressing?
4. How do we compare to others?
One approach is to have industry experts create a very large number of evaluation
questions that cover all aspects of a facility’s design, construction, operation, maintenance and on-going engineering. The questions are organized hierarchically around the
20 elements shown in Table 1.2 and in Figure 14.3.
The topic of Management of Change has been highlighted in Figure 14.3 in order to
illustrate the assessment concept discussed in this section. This topic is divided into 12
sections, as shown in Figure 14.4.
Level 1 – risk management
The top level risk management structure of Figure 14.4 is shown in Figure 14.5 in
spreadsheet format.
The table in Figure 14.5 has four columns. The first column lists the 20 CCPS
management elements starting with ‘‘1. Process Safety Culture’’ and finishing with ‘‘20.
Management Review’’. The topic of Management of Change, which is the 13th of the
elements, is used here to illustrate the spreadsheet process.
The second column shows the total number of questions to do with each topic. In the
case of MOC there are 109 questions. The third column provides the overall score for each topic as a percentage or normalized total. In this example, MOC scores 54%. Each
spreadsheet is given the same weight regardless of the number or nature of the ques-
tions within it.
9. Safe Work
14. Operational Readiness 11. Contractors 15. Operations
5. Outreach
16. Emergencies 17. Incidents 20. Management Review 19. Auditing18. Measurement
12. Training
FIGURE 14.3
Assessment Structure
13.9 Environmental13.7 Procedures 13.8 Asset
Integrity 13.6 Knowledge
Management elements assessment 721
The total number of questions is 1976 and the total score is 61%.
To the left of the spreadsheet the ranking system is explained. A score of <50 is
‘‘unacceptable’’. Therefore any values below 50 are shown in red. A score in the 50–75 range is ‘‘acceptable’’ and is shown in blue. A score of 75 or greater is ‘‘excellent’’ and is
FIGURE 14.5
First-Level Spreadsheet
shown in green. Figure 14.5 shows 4 reds, 15 blues and 1 green. Management of Change
has a score of 54, which is considered ‘‘acceptable’’.
Level 2 – management element spreadsheet
The topic titles in Figure 14.5 are colored blue and are underlined. This means that each
is hyperlinked to a daughter spreadsheet. Figure 14.6 shows the sheet for Management
of Change. It contains 12 sheets, starting with ‘‘13.1 Philosophies’’ and ‘‘13.2 Policies’’,
and finishing with ‘‘13.12 Auditing’’.
The structure of Figure 14.7 is similar to that of Figure 14.6. The number of questions
for each spreadsheet is shown, along with the normalized (percentage) score for each.
Each topic is hyperlinked to another spreadsheet, one level down. Once more a color
scheme showing the ranking for each sheet is used using the same three levels as in Figure 14.6. Of the 14 spreadsheets, 5 have a score of less than 50, which means that the
FIGURE 14.6
result is ‘‘unacceptable’’ – so the score is shown in red. Six have a score in the 50–75
(‘‘acceptable’’) range. Hence they are shown in blue. Just one is above 75. It is ‘‘excel-
lent’’, and is shown in green. The total score is 54% which is ‘‘acceptable’’.
Level 3 – detailed questions
Each of the topics shown in Figure 14.6 has a set of detailed questions associated with it.
Of the 12 sheets to do with MOC, Figure 14.7 provides an example for one of them: 13.3.
Temporary Changes.
At the top of Figure 14.8 is the question ‘‘This sheet to be used?’’ The default condition is ‘Y’. If any other entry is made then the results from this sheet are removed
from the overall scoring system – both numerator and denominator.
Each sheet has a set of questions – generally about 5–15 per sheet. For ‘‘Management
of Change: Temporary Changes’’ there are just five questions. The general philosophy is
that a ‘‘yes’’ answer to a question is desirable. For example, question 13.3.1 asks, ‘‘Is
there a clear definition of what constitutes a Temporary Change?’’ Clearly an answer of
‘‘yes’’ is desirable.
In order to provide a range for the response to a question, a scoring system is provided. The system used here is shown below.
FIGURE 14.7
Third-Level Spreadsheet
0: missing/ineffective
1: poor
2: adequate
3: good
4: complete/excellent
N/A: not applicable
When a range of responses is possible, then an appropriate score is given. With
regard to question 13.3.1, the reviewer may find that the facility does have a definition
for Temporary Change, but that there are some areas of misunderstanding and potential
confusion. Therefore, he/she provides a score of 3. If the question can generate only a ‘‘yes’’ or ‘‘no’’ response, then the score is either ‘‘0’’ or ‘‘4’’. If an answer is not relevant
Template for 13.3.1
Is there a clear definition of what constitutes a temporary change?
Score Discussion
0: missing / ineffective No definition for temporary change exists. 1: poor The definition is insufficiently precise. 2: adequate The definition is clear, but it is not properly
differentiated from other types of change, such as “field change”.
3: good The definition is clear, and covers all normal situations.
4: complete / excellent The definition is clear, and has been validated by other facilities in the company.
FIGURE 14.8
to the current situation then ‘‘N/A’’ is entered; this removes that question from the
calculation of both the numerator and denominator terms in the normalized total.
The individual scores are added together to give a normalized (percentage) total.
Figure 14.8 shows that the overall score for ‘Management of Change: Temporary
Changes’ is 60%.
Scoring template
Figure 14.8 shows that each question number is hyperlinked. The link is to a scoring
template that provides guidance to do with that particular question. Use of the template
helps ensure high levels of consistency between different reviewers and when the
system is being applied to multiple facilities. Figure 14.8 shows a template for question
13.3.1, ‘‘Is there a clear definition of what constitutes a Temporary Change?’’
Guidance
At the top of Figure 14.7 is the word ‘‘Guidance’’. This links to a document that provides
background information and assistance to do with the topic in question – in this case
Temporary Change. Figure 14.9 provides the Guidance that is provided for the topic (it
has been taken from Chapter 13 of this book). Figure 14.9 also includes links to citations
and other reference material.
Benefits of the elements assessment approach
The elements assessments approach has the following advantages over the more
traditional lagging and leading indicator methods.
Independent of events The assessment system does not rely on the occurrence of events or near misses. For
example, even if there has not been an incident attributable to failure of the operational
readiness system, potential problems to do with this element should be identified once the 14 spreadsheets have been completed.
13.3.1 Is there a clear definition of what constitutes a temporary change?
Guidance
Temporary changes are those changes that incorporate within themselves a built-in termination date or time. Changes of this type are often implemented to keep the operation running while a piece of equipment is repaired or replaced.
From a safety and operational point of view, whether or not a change is permanent or temporary is merely a semantic matter — the system itself does not know or care that a change is intended to be temporary. Therefore, the fact that a proposed change is defined as being “temporary” does not mean that it can be handled less rigorously than a change that is intended to be permanent. Yet, because of the short duration of temporary changes, the personnel implementing them may be tempted to take short cuts, particularly if going through the MOC process takes longer than actually making the change itself. There is a temptation to take an attitude of “let’s just get on with it — why bother spending hours writing and reviewing a procedure for an operation that will only take a few minutes to carry out?”
Citations
By contrast, the lagging and leading indicator approaches can only identify under-
performing management elements through the use of root cause analysis. As discussed
in the previous chapter, such analyses are inherently subjective – different people and
different techniques will come up with different answers as to why an event occurred.
Handling abstraction Lagging and leading indicators provide little guidance to do with some of the more
abstract management elements such as process safety culture and management review.
This difficulty is entirely overcome when a management assessment approach is used.
Smoothing of results Through the use of literally thousands of questions, the results are smoothed out. Even if some of the questions are answered incorrectly, the overall results should be of high
quality.
Objectivity The results of the assessment are likely to be quite objective. In the system shown
below, each question has a potential score ranging from 0 (missing or ineffective) to 4
(complete or excellent). It is unlikely that two assessors would come up with dramat-
ically different responses to the questions.
CHAPTER 14 Audits and assessments
INTRODUCTION
Lessons learned
Reviewer attributes
Level 3 - detailed questions
Independent of events

Documents

Chapter 14 - Audits and assessments