Embed Size (px)
Citation preview
Redesign of the SIGA Accident/Incident Reporting System: Providing a Posterior Analysis Function to Identify and Communicate Likely
Future Accidents within the Petroleum Industry
A Thesis in STS 402
Presented to
The Faculty of the
School of Engineering and Applied Science University of Virginia
In Partial Fulfillment
Of the Requirements for the Degree
Bachelor of Science in Systems and Information Engineering
by
John Yandziak
May 9, 2006
Approved ___________________________________________(Technical Advisor) Stephanie Guerlain (Science, Technology Approved __________________________________________ and Society Advisor)
Kathryn Neeley
“On my honor as a University student, on this assignment I have neither given nor received unauthorized aid as defined by the Honor Guidelines for Papers in Science,
Technology, and Society Courses.” ________________________
ii
Preface
The author would like to acknowledge the generous cooperation of the TransPetro subsidiary of Petrobrás Oil Company, specifically those in the Health, Safety, and Environment department at the SouthEast regional branch in Campos Elise. Please note that this is a student paper, created entirely for academic purposes, and opinions and recommendations made therein do not represent in any way the official position of Transpetro or Petrobras. The author would also like to acknowledge the generous funding provided by the CAPES and FIPSE groups in creating the US-Brazil Cognitive Systems Engineering Exchange Program, without which this research would not have been possible. Finally, I must thank my Capstone teammates Osmar de Lima and Monique Verboonen, Industrial Engineering students from the Universidade Federal do Rio de Janeiro (UFRJ), who each contributed just as much to this thesis work as I did myself. Professor Jose Orlando Gomes of UFRJ was the project coordinator for Brazil and arranged all contacts and meetings for Petrobras, even when they were difficult to come by. Professor Stephanie Guerlain was an excellent technical advisor and I want to thank for her patience and guidance as we made it through a full year of American-Brazilian relations together.
iii
Table of Contents PREFACE II TABLE OF CONTENTS III LIST OF ILLUSTRATIONS AND TABLES V GLOSSARY OF TERMS VI ABSTRACT IX
CHAPTER 1 - INTRODUCTION 1
HISTORICAL CONTEXT 1 PROJECT RATIONALE AND BACKGROUND 2 PROJECT GOALS 4
CHAPTER 2 - LITERATURE REVIEW 5
ACCIDENT/INCIDENT REPORTING SYSTEMS 8 DATA OVERLOAD AND INFORMATION VISUALIZATION 9 DATA VISUALIZATION ERROR! BOOKMARK NOT DEFINED.
CHAPTER 3 – HUMAN FACTORS-BASED USABILITY ANALYSIS OF EXISTING SYSTEM 11
DESCRIPTION OF EXISTING SYSTEM 11 SUGGESTED IMPROVEMENTS FOR WITNESS REPORTING 12 LACK OF POSTERIOR ANALYSIS AND STORAGE METHODS 14
CHAPTER 4 - PROPOSED SOLUTION 15
QUANTITATIVE DATA POPULATION – CREATING A NEW WITNESS FORM 16 RELATIONAL DATABASE – ORGANIZATIONAL STRUCTURE 18 DATA VISUALIZATION SOFTWARE – TREEMAPS 20
CHAPTER 5: PROJECT AS SOCIAL EXPERIMENTATION 24
SOCIAL GOALS AND VALUES 24 UNCERTAINTIES, RISKS, AND ETHICAL ISSUES 26
CHAPTER 6: CONCLUSION 29
SUMMARY OF RESULTS 29 INTERPRETATION 29 RECOMMENDATIONS 30 SOCIAL AND ETHICAL ISSUES IN IMPLEMENTATION 31
iv
WORKS CITED 33
WORKS REFERENCED 35
APPENDIX A: FULL SCREENSHOT OF TREEMAPS DATA VISUALIZATION SOFTWARE 37
v
List of Illustrations and Tables
FIGURE 1 : SINKING OF PETROBRAS P-36 PLATFORM........................................................... 2 FIGURE 2 : ANOMALY TYPES FREQUENTLY OVERLAP ....................................................... 13 FIGURE 3 : PROPOSED REGISTRATION FORM FOR WITNESS REPORT ................................. 18 FIGURE 4 : CONCEPTUAL MODEL OF PROPOSED RELATIONAL DATABASE ........................ 19 FIGURE 5 : INITIAL OVERVIEW SCREEN FOR TREEMAPS SOFTWARE.................................. 22 FIGURE 6: FILTRATION SLIDER INTERFACE ...................................................................... 23 FIGURE 7 : TEXT BOX VISUAL CUE REINFORCES INFORMATION ....................................... 24
TABLE 1 : CLASSIFICATION CHANGES................................................................................ 16
vi
Glossary of Terms Accident Analyst – type of user within SIGA system, tasked with examining an accident
report and determining causes of reported accident, then recommending a corrective
action
Analysis Stage – SIGA system hazard remediation stage where accident analyst
identifies root cause of accident or incident and recommends corrective actions for
elimination of present hazard
Approval Stage – SIGA system hazard remediation stage where agency manager views
accident analyst’s recommended corrective actions and approves or rejects said actions
Conceptual Model - depicts the organizational structure of a relational database without
the restriction of specifying certain technologies to implement for the proposed solution
Corrective/Preventative Action – a recommended event to be carried to eliminate the
underlying cause of a reported accident, recommended by the accident analyst
Data Visualization Software – a software that allows the user to view a very large
amount of data simultaneously by only visually displaying descriptive characteristics of
the categorization of the data
Data Filtration – removing unwanted data items from view in order to focus attention on
a smaller, more specific data set
Final Evaluator – type of user within SIGA system, responsible for final approval of a
hazard remediation process; after all corrective actions have been implemented the final
evaluator approves or rejects the entire process
Hazard – a set of conditions that is likely to lead to an accident, or has already lead to an
accident
vii
Hazard Remediation – the entire process implemented by an accident/incident reporting
system, including registration, analysis, approval, implementation, and verification; deals
with the analysis and correction of one event at a time
Heuristic Evaluation – an analysis made by a subject matter expert without the use of a
specific methodology or framework, often uses “rules of thumb”
Implementation Stage – the accident remediation stage in SIGA where corrective
actions are received by field managers and implemented into action
Incident – an event that could have had the same consequences of an accident but did not
actually occur, a “near miss”
Posterior Analysis – an analysis of multiple events simultaneously, conducted after they
occurred to discover developing patterns and trends within the data set
Registration Stage – the accident remediation stage in SIGA where the initial witness
enters into the system his/her account of a accident or incident
Relational Database – a method to organize data as a collection of tables and subtables,
each displayed in columns and rows; traditionally implemented using the Structured
Query Language (SQL) as part of a Relational Database Management System (RDBMS)
RTA – Anomaly Treatment Report (translated from Portuguese phrase Relatório de
Tratamento de Anomalias), collective term used to describe all data related to the
registration, analysis, approval, implementation, and evaluation of an individual
accident/incident report
Safety Culture – a work environment that draws focus away from assigning blame in
order to encourage more accurate accident and incident reporting at a more productive
rate
viii
SIGA – Integrated Anomaly Management System (translated from the Portuguese phrase
Sistema Integrada de Gestão de Anomalias)
Soft technology – compliant, yielding systems that inform, provide rich set of
instructions and options, acknowledge initiative and flexibility of the human user.
Treemap – a data visualization software designed by Ben Shneiderman that organizes
and visually displays data, customized through filters, visual attributes, tree node choice,
Tree node – one of the customizable display variables in the Treemap data visualization
software
Verification Stage – the hazard remediation stage within SIGA where the final evaluate
approves or rejects the validity of the entire Anomaly Treatment Report (RTA)
Visual Attribute – one of the customizable display variables in the Treemap data
visualization software
Witness – the person who watched an accident or incident take place and then enters
their account of the event into the SIGA system during the registration phase of hazard
remediation
ix
Abstract
Petroleum transportation accidents often cause losses of millions of dollars and take
human lives. SIGA (Integrated Anomaly Management System) is an accident/incident
reporting system that was designed by Petrobras, the largest petroleum transportation
company in Brazil, to help remediate these accidents. SIGA’s function is to track
reported accidents and their cost, determine the cause of these accidents, identify
solutions, and communicate accident occurrence and recommended solutions to
appropriate parties. While the existing system administers the workflow of accident
remediation, it lacks the functionality necessary for holistic posterior analysis. Such
analysis is critical and would enable the examination of all reports simultaneously to
detect emerging patterns and trends. This paper proposes a software solution to track
large quantitative data sets composed from incident reports in order to preemptively
identify potential hazards and recommend appropriate actions to eliminate future hazards
before they occur.
1
CHAPTER 1 - INTRODUCTION
This thesis provided an analytical tool for Petrobras, Brazil’s federally owned
petroleum company in order to identify emerging patterns that exhibit high-risk areas,
culled from a database of accident and incident reports, that identify likely causes of
future accidents. This chapter describes the background and motivation for the project
and provides an overview of the project results.
Historical Context
Because Petrobras works within the extremely high-risk petroleum industry, many
disasters mark its history; additionally, as with many disasters, ignorance of safety
precautions and procedures in the pursuit of lower costs and higher profit has been linked
to the root causes of many of these events. For example, a Petrobras official was quoted
in 2000 as proudly declaring the following about the P-36 platform, the largest floating
off-shore oil platform in the world at the time:
“Petrobras has established new global benchmarks for the generation of exceptional shareholder wealth through an aggressive and innovative program of cost cutting on its P36 production facility…The project successfully rejected the established constricting and negative influences of …onerous quality requirements, and outdated concepts of inspection and client control. Elimination of these unnecessary straitjackets has empowered the project's suppliers and contractors to propose highly economical solutions, with the win-win bonus of enhanced profitability margins for themselves.” (Calipso, 2002)
Less than one year later, on March 15th, 2001, the P-36 platform experienced two
unexplained explosions, killing 11 people on board, sinking the platform, and dumping
1500 metric tons of crude oil into the Atlantic Ocean (see Figure 1).
2
Figure 1 Sinking of Petrobras P-36 Platform
Retrospectively, the “win-win bonus of enhanced profitability margins”, stemmed
directly from cost cutting on such “unnecessary straitjackets” as quality assurance and
inspection costs, did not set the “global benchmark” it intended to set. Accordingly,
Petrobras, as well as most prominent petroleum producers, now turn significant financial
and human resources towards new accident prevention programs.
Project Rationale and Background
The US-Brazil Cognitive Systems Engineering Exchange annually recruits
Systems and Information Engineering students at a variety of engineering schools,
including the University of Virginia. The FIPSE-CAPES grant provides the funding for a
group of American engineers to travel to Brazil and study at the Universidade Federal do
Rio de Janeiro while conducting field research at Petroleo Brasileiro S.A. (Petrobras),
Brazil’s federally owned petroleum company. As a Human Factors Engineering design
team, the group was responsible for identifying a health and safety problem within a
3
Brazilian petroleum company and providing a software solution to ameliorate the
identified troubled situation.
With daily processes that involve the exploration, excavation, transportation, and
distribution of highly combustible chemical products, the petroleum industry will always
remain an extremely high-risk industry in terms of the likelihood of large-scale industrial
accidents and the potential severity of the economic and social harms that these accidents
can inflict. A successful accident prevention program must accurately determine the
cause of previous accidents, recommend effect corrective actions to remove the hazards
that caused those accidents, and install a reliable hazard detection system that enables a
company to focus safety efforts in the areas where hazard remediation can have the
greatest effect on future likelihood of serious industrial accidents (Wickens 2004).
Petrobras today already employs a hazard detection software system, entitled
“Sistema Integrada de Gestão de Anomalias” (SIGA)—translated from Portuguese into
“Integrated Anomaly Management System”. SIGA is one of many examples of an
Accident/Incident Reporting System—safety information systems utilized by large
companies to monitor, administer, and analyze the large amount of descriptive data
obtained from eyewitness accounts of accidents and incidents. (Experts typically refer to
events that actually occurred and caused some form of harm as accidents, while an
incident, is defined as a “near-miss”, which could have resulted in a negative occurrence,
but did not) (Wickens 2004). The system has been in existence since 2002, and in the
four years since its creation has registered over 10,000 reports. (Personal Communication
with Petrobras official 2006).
4
Project Goals
This project conducted a critical analysis of the current SIGA system in order to
identify areas of improvement that could be addressed in a proposed software redesign.
After analysis, the project team decided to propose a posterior analysis function that will
identify emerging trends of hazardous behavior within an organization, determined from
the cumulative history of previous accidents and near-misses, in order to identify critical
hazards that are prevalent throughout the organization so that may be addressed on a
company-wide level by top-level management. Posterior analysis involves looking at
large numbers of reports simultaneously in order to discover emerging trends that are
existent across many areas of the data set.
This project seeks to redesign the current SIGA Accident/Incident Information
System in order to provide the following additional features:
1. A new witness reporting form that allows for more accurate, robust incident
accounts that are more readily analyzed in a quantitative manner;
2. A relational database to keep track of the resulting large amounts of new
quantitative data; and
3. A data visualization software that enables users to identify patterns within the data
set that point to critical areas within the company where safety resources can be
most effectively leveraged.
5
CHAPTER 2 - LITERATURE REVIEW
This chapter, reviewing relevant literature, seeks to answer the question: What
concrete effect can an improved Accident/Incident Reporting System, with the addition
of a Posterior Analysis function, have on the rate of future accident occurrence in the
petroleum industry?
Safety Management Programs and Their Evaluation
Safety management is a holistic and systemic approach that attempts to identify
hazards that lead to accidents and eliminate their causes before they lead to accidents. A
systemic approach views accidents through the perspective that they are caused by no
single factor or error, but a combination of factors that interact with each other.
A successful safety management program should be developed by management
with significant staff input, because employee involvement in development has been
proven to improve acceptance and effectiveness of a new system (Robertson et al., 1986).
The implementation involves three stages: risk identification through examination of
company documents such as accident/incident reports, safety records, and training
materials, the development and execution of a safety program that includes
accident/incident reporting, safety rules and procedures, safety equipment use, and
employee training, as well as a evaluation program that determines program
effectiveness through quantitative measures such as number of casualties, monetary loss,
or days off due to injury. (Manning, 1996).
Most organizations rely on such negative outcome measures to evaluate success
of their safety programs. However, Reason finds this data an unreliable indicator of a
system’s “intrinsic safety”. Many safety systems have become successful enough in
6
accident prevention that these outcome events now occur so infrequently that analysis
between periods becomes difficult, as any small fluctuations bear little more than useless
“noise”, in terms of statistical analysis (Reason, 1997). An example is the aviation
industry where the worldwide risk has remained constant, at a value of one passenger
fatality for every million air miles, for over 20 years, despite significant advances in
safety procedures (Howard, 1991). Reason therefore suggests a switch to process
measures as indicators of the overall health of an organization’s safety system, which can
be evaluated as the position within Reason’s “safety space” framework, depicted in
Figure 2.
Figure 2 – James Reason’s “Safety Space” [Created by Author, adapted from Reason, 1997]
The goal of an organization in the Safety Space evaluative framework is to constantly
push itself towards the side of “increasing resistance”, though “countervailing currents”
constantly push organizations towards average, in between resistant and vulnerable.
7
Reason contends that Mintzberg’s 3 C’s framework is the “safety engine” that drives
organizations towards maximum resistance (Reason, 1997).
Mintzberg’s Three C’s – The “Safety Engine” Mintzberg states that the three driving forces that must make up the “strategic
apex” of any effective safety management system are commitment, competence, and
cognizance. Commitment includes the organizations motivation to be an industry leader
in establishing a strong safety culture and the human and financial resources they are
willing to commit to that end. Competence is very closely related to the quality of an
organization’s safety information system, and can be evaluated through the system’s
ability to collect the right information, to disseminate that hazard information warning
properly (to the right people in a timely fashion and in a presentable method), and, after
the hazard data is received, to act upon the information to produce concrete results.
(Mintzberg, 1989).
Studies comparing effectiveness across multiple organizations concur that top-
level commitment and possession of an adequate safety information system are the top
two characteristics most likely to distinguish safe organizations (Smith et al., 1988).
Without the first two identifying characteristics, the third, cognizance, would not be
possible; though it should be considered equally as important. Cognizance means the
basic situational awareness that an organization has of the critical hazards it currently
faces, and a successful accident/incident information system that communicates critical
hazards to the appropriate parties is the key to this situational awareness.
8
Accident/Incident Reporting Systems
Accident/Incident Reporting Systems are so crucial to hazard management
programs in high-risk companies that regulatory agencies have begun to require,
standardize, and monitor their use in certain industries. The Occupational Safety and
Health Administration (OSHA), under the U.S. Department of Labor, regulates many
high-risk industries, including the petroleum industry, through its published safety
standards (OSHA, 2006). OSHA standards require accident reporting and investigation
for all of its regulated industries, and for the petroleum industry specifically, the
administration also mandates collection of incident reports in addition to accident reports,
through OSHA rule 29 CFR1910.119 (Wickens, 2004).
Many systems still do not include incidents in their reporting systems, only
accidents; for example, SIGA itself did not include them until September 2005 (Personal
Communication with Petrobras official, 2005). However, any effective safety
information system should consider the inclusion of incident gathering as a component
due to its many advantages, particularly those in quantities of data collected. Reason
considers incidents as “free lessons”, primarily for three reasons (Reason, 1997):
1.) They provide insight into how small defensive failures add up to large disasters,
2.) They occur more frequently than accidents, yielding larger numbers of data
required for penetrating qualitative analysis,
3.) They provide a powerful reminder of system hazards to communicate to top-level
management
In high-risk industries, where potential hazard consequences can be severe,
incidents produce the extra amount of data required to obtain a clear picture of potential
9
hazards and their causes and communicate that danger to top-level management.
However, the quality and amount of data depends on the willingness of individuals to
report witnessed accidents, which is often a challenge if the witness is the same person
that committed the offense. The solution is to create a “reporting culture” where
individuals can report without fear of blame. The Aviation Safety Reporting System can
be looked to as an example in creating an environment where witnesses are encouraged,
not discouraged, to report.
The Aviation Safety Reporting System (ASRS) is a voluntary, confidential, and
non-punitive incident-reporting system that collects and disseminates information among
such varied sources as airline pilots, maintenance technicians, the Federal Aviation
Aviation (FAA), and the National Aeronautics and Space Association (NASA). Before
its creation in 1974, accident and incident reporting as well as reprimand and regulation
for committed accidents were handled by the FAA, creating a conflict of interest that
scared away pilots from reporting near-misses for fear of punishment. The establishment
of the ASRS created an independent, objective system, run by NASA apart from the
FAA’s punitive responsibilities, left only in charge of near-misses, or incidents.
(Accidents, where there can be blame assigned, are now investigated separately by the
National Transportation and Safety Board). Pilots are now encouraged to submit incident
reports, knowing that their information will be fully analyzed to prevent future accidents
and improve aviation safety, and not to distribute blame for wrongdoing (Billings, 1999).
Data Overload and Information Visualization In any accident/incident reporting system, and especially in those where incident
reporting is encouraged, there exists a prohibitively large amount of data to be scrutinized
10
in order to find relevant and important information. In examining data, users have
limited attention and must deploy that attention in a way that is adaptive to the task of
finding valuable information (Resnikoff, 1989). A situation therefore develops where
“the scarce resource is not information; it is the processing capacity to attend to
information,” (Simon 1976). The challenge of “data overload”, defined as “finding what
is informative given our interests and needs in a very large field of available data”
(Woods et al., 2002). The most effective way to find what is informative in very large
data sets is data visualization, or providing a method to use visual intelligence to discover
patterns; if visual patterns are made easy to perceive, “we facilitate problem solving”
(Ware, 2003).
Users, when visually presented data in the appropriate manner, are naturally very
skilled at directing their attention to the areas of the perceptual field that are of high
relevance given the properties of the data field and the expectations and interests of that
user (Woods, 1984). The key to solving the data overload challenge is therefore to make
sure that the properties of the viewed data field and the expectation of the user point the
user to information that is actually informative and relevant to his or her goals of use.
Therefore, the massive amounts of reports generated in an accident/incident reporting
system can be effectively visually analyzed if a data visualization software presents that
data in the right way to the user, as this thesis’ proposed software solution seeks to do.
11
CHAPTER 3 – HUMAN FACTORS-BASED USABILITY ANALYSIS OF EXISTING SYSTEM
An in-depth analysis based on Human Factors Engineering methodologies was
conducted in order to identify confusing or clumsy parts of the human-computer interface
associated with the SIGA Accident/Incident Reporting System.
Description of Existing System
The system under review, SIGA, is operated and maintained through Transpetro’s
Health, Safety, and Environment department. This Accident/Incident Information System
solution is administrated entirely through Lotus Notes, a commercially available client-
server collaborative software system.
The current SIGA design is used primarily to administrate the work flow involved
with the submission, analysis, and subsequent remediation of an accident or incident
report. Access to the system is granted to all employees of the large petroleum provider,
as well as all subcontractors currently participating in a project with the petroleum
provider. System administrators grant access to SIGA only after successful completion
of a short online training program.
The wide majority of the SIGA system users are witnesses, who submit an initial
report including the details of their own personal account. Once a witness report is
submitted, an administrator reviews the report for inaccuracies or missing information,
and then refers the report to the appropriate accident analyst. Typically, the analyst is an
employee within the Health, Safety and Environment department with a specialization
that is related to the referred report’s subject matter. This person analyzes the cause of
the event and recommends corrective actions. From there, agency managers are assigned
responsibility for actions recommended by the analyst and confirm their successful
12
completion to the system once the actions are implemented to satisfaction. Finally, after
all recommended actions are completed, the entire report is forwarded to the final
evaluator for his or her approval. If the reported hazard is not deemed sufficiently
corrected after the final evaluation, the report is re-entered into the system, and the
process starts over from the analysis stage.
Each of the five different user types participates in its own respective stage of the
report remediation process. Witnesses conduct registration of the report; the accident
analysts recommends corrective actions in the analysis stage; recommended actions are
evaluated and permitted in the approval stage; the agency managers administer
implementation of recommended actions; and finally, the final evaluator oversees the
verification stage. The bulk of this project’s critical review and redesign process will
focus on the registration stage, which centers on how information is gathered and
organized, as opposed to the other four stages, which serve largely administrative
functions.
Suggested Improvements for Witness Reporting
Much of the responsibility of information quality falls on the witness, although
often they are not qualified or willing to produce the amount of information needed to
conduct a thorough accident analysis. Adams and Hartwell believe that the level of skill
required to file a report is a function of the quality of information required and of the
design of the report. In their study, they found that only a minority of those making
reports possessed sufficient training and ability to make comprehensive reports and few
plants had made provision for training (Adams and Hartwell 1977).
13
While filling in such standard fields as location, date, descriptive event details,
and his or her unique employee identification number causes little confusion, the current
witness report form requires an additional, more complex analysis that involves the user’s
own subjective evaluation abilities. These additional fields include anomaly type, used
for organizational purposes, possible consequences of the reported situation, and, finally,
fields to evaluate the gravity, likelihood, and incidence of the event.
The current options available to the user for anomaly type—unexpected result,
operational failure, accident, accident with injury, abnormal occurrence, danger,
nonconformity, third party complaint, and other—are vague and difficult to distinguish
because of frequent overlapping (see Figure 2),
Figure 2: Anomaly Types Frequently Overlap [Adopted and translated by Author from Petrobras company document]
A new set of anomaly type categories must be developed where each distinction is
uniquely meaningful. Possible consequences should also be limited to a set number of
options, as opposed to its current blank form state, in order to produce more matching
between separate reports and less confusion in reporting.
14
Whereas with the “anomaly type” and “possible consequences” entry fields the
user had too much choice, there also currently exist multiple categories where the user is
not given enough choice. The gravity option is limited to a choice between large and
small, the likelihood limited to real or potential, and the incidence to initial or
reoccurring. Though the witness may, at times, not be fully qualified to make an accurate
evaluation, if s/he can only distinguish between two levels of granularity, the information
provided is insufficient to conduct any type of data filtration when a posterior analysis is
conducted. To quantitatively evaluate the likelihood of a potential hazard, we
implemented a 1-5 scale corresponding to frequent, probable, occasional, remote, and
improbable, adopted from a previous example (Roland and Moriarty, 1990). Likewise,
following the Department of Defense’s hazard-assessment matrix example, severity can
be evaluated on a 1-4 scale representing catastrophic, critical, marginal, and negligible.
The multiple of these two quantitative measures forms a “criticality” variable, an
excellent quantitative indicator for the elimination priority of a particular hazard
(Department of Defense, 1984).
Also, there currently exists no functionality to allow for multiple witness reports of the
same event. One witness is often not willing, for fear of receiving blame, or not able, for
lack of situational awareness, to provide a comprehensive account of the witnessed event
(Reason, 1997). A more accurate account of any event should include multiple
perspectives blended into one holistic account that blends characteristics from each.
Lack of Posterior Analysis and Storage Methods
As the system currently exists, reports are dealt with entirely on a one-by-one
basis, ignoring the possibility for posterior analysis of many reports at once. The goal of
15
the current SIGA system is only to administrate the hazard remediation process, report by
report, to ensure that once a hazard has been reported, appropriate corrective actions are
implemented to ensure its elimination. However, corrective actions cannot always
completely eliminate a hazard entirely, especially because actions usually only occur at
the site where the hazard was identified. If a function was in place to identify critical
hazards that occur frequently across many work sites company-wide, the organization
should be able to identify this pattern and communicate corrective actions to all affected
branches and agencies within the organization.
The SIGA system’s current storage methods encumber any type of analysis across
many reports, including even the most basic search functionality. Typically, data sets of
such large numbers are stored within relational databases, where quantitative data is
stored in columns and rows that is easily queried to quickly and effectively find desired
information (Connolly and Begg 2005); however, SIGA’s current Lotus Notes platform
contains no relational database storage method, largely due to the fact that there is little
quantitative data to store in the first place. The following chapter discusses the proposed
software solution package to implement a posterior analysis function customized for
SIGA’s accident/incident reporting system that addresses the current system’s need for a
better, more quantitative categorization method combined with a relational database
management system to store and organize that quantitative data.
CHAPTER 4 - PROPOSED SOLUTION The SIGA system’s needs for better categorization, a better storage method, and a
posterior analysis function were each addressed in the proposed software solution
package, consisting of a new witness reporting form to better categorize inputted
16
information, a proposed conceptual model for a relational database structure to better
organize that information, and a data visualization tool to enable posterior analysis of
many reports simultaneously to detect emerging patterns within the data set that point to
critical hazards that occurring frequently across the organization.
Quantitative Data Population – Creating a New Witness Form
In the proposed solution, the previous witness report form and its vague queries
that were not conducive to quantitative analysis were replaced by new fields that require
more specific information, either through the use of pull-down menus with limited
options, or through quantitative evaluation scales that encompass a wider range of
choices to the user.
Table 1 : Classification Changes
Report Field Previous Classification New Classification
Anomaly Type
Unexpected result, operational failure, accident, accident with injury, abnormal occurrence, danger, nonconformity, third party complaint, and other
Hardware, design, maintenance management, procedures, error-enforcing conditions, training, and defenses
Possible Consequences None. (Subjectively determined by user)
Injury, death, financial loss, unsafe company culture, damage to organizational reputation, environmental harm, and damage to organizational property
Likelihood Real or potential Frequent, probable, occasional, remote, or improbable (1-5)
Severity Great or small Catastrophic, critical, marginal, or negligible (1-4)
Criticality None. (Did not exist) Criticality = likelihood indicator * severity indicator
17
Incidence Initial or reoccurring First time, second time, third time, or more than three times
“Anomaly type” and “possible consequences” were identified as fields within the
current witness report form that demanded a subjective analysis from the user that
produced an array of answers that were too disparate to be able to organize in any
meaningful way. Through the use of a behavior-shaping constraint, a system can lead a
user to produce only desirable outcomes (Vicente 2000). Our proposed solution
implemented a drop-down menu as a method to constrain the user’s possible choices in
describing anomaly type and possible consequences. Anomaly type was changed to a
drop-down menu consisting of eight options—hardware, design, maintenance
management, procedures, error-enforcing conditions, training, and defenses. These new
categories were adopted from the Eleven General Failure Types used by the Royal/Shell
Group petroleum company within “Tripod-Delta”, their own accident/incident
information system developed under Jame Reason’s guidance (Shell, 1997).
For possible consequences, the proposed new report form limits the user’s choices
to the following seven categories (from which the user can choose one or more): injury,
death, financial loss, unsafe company culture, damage to organizational reputation,
environmental harm, and damage to organizational property. An entry box still remains
for the user to provide a further description of what may occur, but by forcing the user to
categorize possible consequences, the data will be more easily organized and filtered
when using the data visualization software.
Contrary to limiting a user’s options when he/she has too many, for “likelihood”,
“severity”, and “incidence”, we increased the number of his/her choices because there
18
were too few. As mentioned before, likelihood had its scale changed from “real” and
“potential” to a 1-5 scale (frequent, probable, occasional, remote, and improbable) and
severity changed from “high” and “low” to a 1-4 scale (catastrophic, critical, marginal,
and negligible). In a similar fashion incidence was modified from “initial” or
“reoccurring” to a choice of first time, second time, third time, or more than three times.
The new proposed registration method addresses the lack of multiple perspectives
by adding the functionality for reports from multiple witnesses about the same event to be
submitted and categorized together. The registration report form will identify if the
values entered in the new report match any existent report. If it doesn’t match any other
report the ball remains green, otherwise it changes to red (see Figure 3). The form also
shows the number of reports that match. This functionality is made possible through the
additional organizational capabilities our proposed relational database will provide.
Figure 3: Proposed Registration Form for Witness Report
Relational Database – Organizational Structure
The addition of a relational database management system brings an organizational
capability not available in the current SIGA system. Figure 4 depicts the conceptual
model of our proposed relational database. A conceptual model depicts the structure of
19
the system without the restriction of specifying certain technologies to implement for the
proposed solution (Connolly and Begg, 2005). The conceptual model diagrams the
processing of Relatório de Tratamento de Anomalias (RTAs)—translated from
Portuguese to mean Anomaly Treatment Reports, this is the term used to describe all data
related to the registration, analysis, approval, implementation, and evaluation of an
individual accident/incident report.
Figure 4 : Conceptual Model of Proposed Relational Database
20
It is important that the database only store one report per anomaly or merge
different reports of the same anomaly to have a more detailed view of what happened. So,
the user has the option to create a new RTA or to add a description in the Registration
section of an existing RTA. When the user chooses the “create a new RTA” option, after
the user finishes the registration section, the system compares this report to other reports
in the database, searching for RTAs for the same anomalies that might have been created.
This is important because it is possible that the user doesn’t know about the existence of
these reports. This is the feature displayed in the previous example, Figure 3, by the red
and green display buttons.
When searching for identical reports, the system compares specifics fields: title,
managing agency, anomalies type, location and a range of date. If it finds RTAs that
match those fields, the system presents the user with a list of these reports. The user must
then check if any of them are reporting the same anomaly. If so, he can either merge the
two reports or add his report as a second version of the first one. In case he decides to
merge them, the system excludes some fields (such as managing agency, anomaly type,
location of occurrence, anomaly severity, likelihood, and incidence) and maintains the
sections with the description of the anomaly, its possible consequences and the
immediate actions in order to provide an additional account of the events that transpired.
Data Visualization Software – Treemaps
While the proposed solution includes the new witness reporting form in order to
categorize data better, and the relational database in order to organize that data, the data
visualization software represents the true power of our proposed software package—
analyzing our data in its new categorized and organized form in order to conduct a
21
posterior analysis. The goal of the posterior analysis is to view all of the reports
registered within in the system simultaneously to discover trends and patterns that
indicate where the hazardous areas exist within the organization. This goal is normally a
challenge because the sheer amount of data that must be analyzed cannot be accurately
represented on a computer screen without creating a data overload problem for the user.
To combat the data overload problem, the project team incorporated a SIGA-
customized version of Shneiderman’s Treemaps data visualization software with our own
relational database storage system. The widely cited visual-information-seeking mantra
states that successful information visualization is conducted through a recursive
exploration described as “overview first, zoom and filter, then details on demand.
Repeat.” The design team chose Shneiderman’s TreeMaps software because it
incorporates an initial overview with a movable field-of-view box, user-controlled zoom
factors, dynamic queries in the form of sliders that allow the user to filter uninteresting
items according to a indicator variable, and drilldown capabilities which capacitate quick
detail reference on any selected item (Shneiderman 2005). With these features a user can
obtain a quick overview of the data in its entirety, zoom in on a specific area and/or filter
out unrelated information based on his/her search preferences in order to identify a
hazardous area, and finally use the drilldown capabilities to view the specific reports that
they are interested in within that specified hazardous area.
In our own SIGA-customized version of the TreeMaps software, there are specific
categories of tree nodes, filters, and visual attributes that combine to produce a dynamic
query environment that visually reveals patterns and trends from a very large set of data.
The tree nodes in our software are location of occurrence, responsible agency, and
22
incident category, and the visual space in our initial overview screen is divided according
to these nodes, as seen in Figure 5. In this example, N/NE (the North/Northeast region)
is the location under review (reached after the user has zoomed in to this area),
responsible agencies are subcategories that include water terminals, gas pipelines, EHS,
and oil pipelines, and under each subcategory is the incident category defining the
anomaly cause, such as communication or training.
Figure 5 : Initial Overview Screen for Treemaps Software
Within the visual space, the information displayed can be filtered down through
the choice of any or all of the following variables: likelihood, gravity, criticality, and
number of reports, simply by manipulating the sliders on the right sidebar, shown in
Figure 6.
23
Figure 6: Filtration Slider Interface Finally, the visual attributes displayed within the visual space – label, size, color –
can each also be customized to represent the same variables (likelihood, gravity,
criticality, and number of reports). The labels for each rectangle additionally are able to
display the name of the variable displayed. From the example in Figure 5, color has been
chosen to represent criticality, size to represent number of reports, and label to represent
name. By quickly examining this visual space, a user can determine that communication
is a problem area within the Water Terminals agency in the N/NE region, because the
large rectangle denotes a large number of reports, and the green color denotes a high
24
average criticality of those incidents, this determination is further reinforced by a text box
visual cue displayed when the user mouse over any specific box, as seen in Figure 7.
Figure 7 Text Box Visual Cue Reinforces Information
These features combine to provide a powerful software solution for an analyst to detect
hazardous trends quickly and efficiently over thousands of reports at a time by allowing
the user to quickly drilldown to the information he/she deems most important.
CHAPTER 5: PROJECT AS SOCIAL EXPERIMENTATION
When viewed in terms of its social goals and values along with the uncertainties in its
realization, this project can be considered an example of social experimentation.
Social Goals and Values
An implemented software solution added to SIGA’s current interface will affect how
many people in Brazilian society live, work, and think. The job of the people of the
Health, Safety, and Environment department at Petrobras is to save human lives, and this
software assists them in that goal. At the same time, implementation consumes financial
and human resources, and like any new project, Petrobras management must determine if
25
the cost is worth the added benefit.
While analyzing the cost of our proposed solution is rather straightforward,
determining the actual benefit is difficult. Negative outcome measures can be analyzed in
terms of financial cost, for example it has been calculated that each workplace fatality
costs United States society $780,000 per victim. However, as mentioned in discussion of
Reason’s “safety space” framework, a safer organization does not necessarily lead to less
negative outcomes. Because of the relatively infrequent occurrence of large industrial
disasters and the random nature of their occurrence, it is possible that unsafe
organizations can be lucky and avoid any major occurrences, while, at the same time, a
safe organization can be unlucky and have a disastrous event occur. The question,
therefore, remains: do financial resources spent on accident prevention lead to a direct,
measurable benefit?
Reason contends that though safety health of an organization may not be easily
determined through consideration of negative outcome measures – man hours of work
lost, fatalities, injuries, etc. – these measures should not be the evaluation criteria for a
safety management system, anyway. Instead, an organization should strive to reduce
their vulnerability to accidents at all times, and push themselves to the outer edge of the
safety space. A direct benefit of this is obtaining a favorable reputation within your
industry as a safety leader, which can attract new customers and appease employees.
Amartya Sen won the Nobel Prize in Economics for making a similar argument,
but on a much more global level. Just as the safety health of an organization should not
be measured by standard quantitative indicators that are currently the norm, Sen believes
that neither should development be viewed solely in terms of standard economic
26
indicators such as Gross Domestic Product (GDP). Instead, in his book “Development as
Freedom”, Sen opines that freedom itself—in the forms of political freedom, economic
facilities, social opportunities, transparency, and security—is both constitutive of
development and instrumental to that development at the same time. This means that
these forms of freedom should be the goal of the development of a society, not simply
economic growth; while at the same time advancing towards the goal of freedom will
also fuel further development, as a greater number of healthier and better educated people
have more opportunities to contribute in meaningful ways once free of oppression. (Sen
1999). As this software is designed to be implemented in a developing country that was
under the oppression of a military dictatorship as recently as 30 years ago, any positive
effect on Brazil’s development will lead towards greater individual freedoms for its
citizens.
Uncertainties, Risks, and Ethical Issues
Because this project is a social experiment, the results are still unknown and we
should therefore plan for monitoring and maintenance in the case of unintended
consequences. Unintended consequences could include the rejection to new system
implementation by users, original system designers, and/or top-level managers.
Users of the current system that must convert to using our new proposed system
will be the group most affected by the proposed change, and most at risk for a possible
rejection. Users become comfortable in their ways after using one software system for a
long period; even if the system is ineffective, they can learn workarounds for the bugs or
deficiencies that may be existent. Learning new software takes a considerable amount of
time and effort, and those that have become experts on the previous system will lose
27
utility of their previously marketable skill. Informed consent is important with the user
population so that sufficient training convinces the users of the benefits of the new
system and educates them enough to effectively use its powerful new benefits to their
advantage.
System developers face the challenge of implementing the new system that must
cater to that user population; meanwhile, they may not be interested in the system at all
because it is replacing their own work. Cross-cultural adaptivity is always a concern in
software interface design. Seeming simple interface choices, such as color, can be
construed differently across different cultures (e.g. the color red can mean “danger” to
one, but “good fortune” to another) (Ware, 2003). Only the local Brazilian developers
will have the cultural understanding necessary to implement our American solution in a
Brazilian work environment; however, precisely because the proposed solution is foreign,
those developers may be reluctant to accept it in the first place. In essence, a redesign
based on a critical review of the system that they themselves created signifies that their
original design was wrong, and there is a risk that they are not willing to accept that fact.
Finally, managers will be wary of the large financial and human resource cost
involved with this new project. Weighed against potential benefits, it is difficult to
justify, because an effect on typical quantitative negative outcome measures will be
difficult to evaluate. Even if concrete benefits are sufficiently demonstrated, the SIGA
system is only 3 years old, and a good deal of money and resources have already been
committed to the current system, making it more difficult to start over with a new one.
Also, if there exists no safe exit to return back to his original legacy system, the manager
faces another substantial risk of losing a great deal of information and being left with no
28
functional safety information system at all. Already, the new system has no functionality
in place to recover the 10,000 reports already entered; if the manager has to lose 10,000
more reports 3 years from now, he may well lose his job as well.
The best method to manage risk over time for all concerned stakeholders in the
system is to methodically implement the software solution in stages, monitoring progress
made and acceptance of the system by users, system developers, and managers while
obtaining continual feedback as to what improvements can be made. Additionally, to
avoid risk of litigation following a deadly accident, all reports should be made
anonymous or confidential to protect those involved. The implementation of a new
safety information system has its involved risks, but without the realized improvements
in safety, the alternative is a far more precarious proposition.
29
CHAPTER 6: CONCLUSION
Summary of Results
Through the combination of a reporting culture that encourages employees to
enter incident reports without fear of blame to provide sufficient data, an easy-to-use and
meaningful reporting form to maintain data quality, a well-organized relational database
to store the data, and an information visualization software to view large amounts of data
at the same time and identify patterns, users of our proposed system should be able to
identify more potential hazards, identify which hazards are most critical, and
communicate the presence of those high-priority hazards in order to successfully apply
corrective actions that eliminate not only hazards, but their underlying causes as well.
Interpretation Though the project team produced an excellent prototype that is capable of
achieving our goal to identify high-risk behavior trends across an organization and
communicate them effectively, we do not know if the proposed solution will experience
the same success in Brazil. Further user testing must be conducted with actual Brazilian
Petrobras employees that have used the old system and would be using any proposed new
system. The system still must be translated into the Portuguese language, and cultural
considerations must be taken into effect that can only be realized after the software has
been tested within a Brazilian work environment.
In terms of incorporating data from the previous system, as the proposed
implementation plan, described in the next section, currently stands, there is no intent to
enter prior incident reports into the current database, because of the amount of manual
labor required for this. If a software application can be produced to accomplish the
30
automatic translation of previous reports into quantized database relations, then the new
system could account for all of the previous reports entered. If this application is not
made available, it is possible that Petrobras would be reluctant to accept any new system
that does not incorporate the data collected from its legacy system.
Recommendations
In the next year, work on this project will be continued by a similar Capstone
team with the support of US-Brazil Cognitive Systems Engineering Exchange Program.
The next steps to be carried out by future academic teams can include further user testing,
development of a more functional prototype, implementation of a prototype within a
work environment, and a usability analysis conducted on Brazilian workers using our
proposed software solution. In the design of any good user interface, these user tests must
be included along all iterations of the design process, from initial prototyping to
redesigns, through to the final product (Wickens, 2004). Proposed user testing,
prototyping, and redesign should therefore be continually iterated, whether by an
academic project team or later on when Petrobras or another company hopefully
continues the project commercially, until an acceptable final product is produced.
A robust and accurate safety information system that collects the right data,
identifies the most pertinent hazards, and disseminates that hazard information to the
right people at the right time is the key to any successful safety management program.
Collecting the right data requires a simple witness registration for entering reports,
already provided with our proposed solution, accessed and used within a work
environment that practices a safety culture that does not scare the user away from
submitting witness reports. A successful implementation therefore must embrace a safety
31
culture that encourages reporting, a management program that seriously considers
recommendations made by the program and its users, while suggesting effective solutions
that eliminate root causes of hazards, and not just the hazards themselves (Reason 1997).
In order to produce a safety culture that encourages abundant reporting of
incidents, an effective training program with a renewal requirement every year would
raise the average ability level of the typical user as well as encourage more users to buy
into the system. Anonymous reporting or guaranteed protection of anonymity once
reports are entered, along with immunity from punishment (and perhaps even rewards for
reporting) are also required elements to encourage reporting. The Aviation Safety
Reporting System and its voluntary, confidential, and non-punitive reporting system
should be looked to as an example.
Social and Ethical Issues in Implementation
In implementing a complex new technology on a large-scale in an
underdeveloped country, one must consider whether the country has the social structure
to receive it. Canadian journalist John Stackhouse provides a telling example of an
Indian couple that uses up a doctor’s year-long prescription for birth control pills in only
6 months. An inquiry into the matter revealed that both the husband and wife had
dutifully taken a pill everyday, because they had never received proper instructions on its
use because the doctor assumed it was common knowledge that only the female required
the pill (Stackhouse, 2000). Vicente believes that “human development can’t be viewed
as a ‘plug and play’ activity where technology is airlifted from one part of the world to
the other. Context matters.” A special attention must be paid to the “soft” technogical
systems that aid the social structure to adapt complex new technologies (Vicente 2000).
32
Soft technology can be described as compliant, yielding systems that inform, provide rich
set of instructions and options, and acknowledge the initiative and flexibility of the
human user (Norman, 1993). In summary, our software should be designed with the user
experience in mind, and not vice versa; this can only be accomplished by including
human user testing in all future phases and iterations of our software design.
Technology must be particularly flexible because of cross-cultural considerations.
The redesign of the SIGA interface done in this thesis report was conducted entirely here
in the United States and in the English language. If this proposed redesign is
implemented, it will happen in Brazil and the software interface must be translated into
Portuguese. Users of the SIGA system include any worker that participates with
Petrobras in any form, across the entire country, and this population will include a diverse
cross-section of socio-economic backgrounds, education levels, and computer literacy
abilities. The implemented interface should therefore be as simple and easy to
manipulate as possible, and I believe our proposed redesign, when compared to the
current system in place, is a step in the right direction.
33
WORKS CITED
Adams, N. L., and Hartwell, N.M. “Accident-Reporting systems: A Basic Problem Area in Industry. Journal of Occupational Psychology, 50. (1977): 285-298. Billings, Charles. The NASA Aviation Safety Reporting System: Lessons Learned From Voluntary Incident Reporting. Enhancing Patient Safety and Reducing Errors in Health Care. Chicago: National Patent Safety Foundation, 1999. Calipso Consulting. "Interesting News." Caliso Worldwide Consulting and Training. June 2002. 28 Apr. 2006 <http://www.caliso9000.com/interestingnews.html>. Connolly, Thomas, and Carolyn Begg. Database Systems: a Practical Approach to Design, Implementation, and Management. 3rd ed. Essex, England: Pearson, 2005. Howard, R W. Breaking Through the 106 Barrier. International Federation of Airworthiness Conference, Oct. 1991. Auckland, New Zealand, 1991. 20-23. Kohn, T. P., M. A. Friend, and C. A. Winterberger. Fundamentals of Occupational Safety and Health. Rockville, MD: Government Institutes, 1996. Manning, M. V.. So You’re the Safety Director: An Introduction to Loss Control and Safety Management. Rockville, MD: Government Institutes, 1996. Mintzberg, H. Mintzberg on Management: Inside Our Strange World of Organizations. New York: The Free P, 1989. Norman, Donald A. Things That Make Us Smart. Cambridge, MA: Perseus Books, 1993. 232-233. Reason, James. Managing the Risk of Organizational Accidents. Brookfield, VT: Ashgate, 1997. Resnikoff, H.L. The Illusion of Reality. New York: Springer-Verlag. 1989. Robertson, G.G., S. K. Card, and J.D. Mackinlay. “Information Visualization Using 3D Interactive Animation.” Communications of the ACM. 36(4). 1993: 57-71. Roland, H, and B Moriarty. System Safety Engineering and Management. 2nd ed. New York: Wiley, 1990. Sen, Amartya. Development as Freedom. New York: Anchor Books, 1999. Shell. Review of Tripod-DELTA. Aberdeen: Shell Expo UK, 1997, 17-18.
34
Shneiderman, Ben, and Catherine Plaisant. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 4th ed. College Park, MD: Pearson Addison Wesley, 2005. Simon, H. A. Administrative Behavior. 3rd ed. New York: Free Press. 1977. p. 294-295. Smith, M J., H Cohen, A Cohen, and R J. Cleveland. "Characteristics of Successful Safety Programs." Journal of Safety Research 10 (1988): 5-14. Stackhouse, J. Out of Poverty and into Something More Comfortable. Toronto: Random House Canada. 2000. "The OSHA Homepage." Occupational Safety and Health Organization. 7 Apr. 2006 <www.ohsa.gov>. United States. Department of Defense. M.L-STD-882B. Washington D.C.: U.S. Government Printing Office, 1984. Vicente, Kim. The Human Factor: Revolutionizing the Way People Live with Technology. New York: Routledge, 2004. Ware, Colin. "Design as Applied Perception." HCI Models, Theories, and Frameworks: Towards a Multidisciplinary Science. Ed. John M. Carroll. San Francisco: Morgan Kaufman, 2003. 16-26. Wickens, Christopher D., John D. Lee, Yili Liu, and Sallie E. Becker. An Introduction to Human Factors Engineering. 2nd ed. Upper Saddle River, NJ: Pearson Prentice Hall, 2004. Woods, D D., E S. Patterson, and E M. Roth. "Can We Ever Escape From Data Overload?" Cognition, Technology, and Work 4 (2002): 22-36. Woods, David. “Visual Momentum: A Concept to Improve the Cognitive Coupling of Person and Computer.” International Journal of Man– Machine Studies. 21 (1984):229–244.
35
WORKS REFERENCED Blackburn, Simon. Being Good: A Short Introduction to Ethics. Oxford: Oxford University Press. 2001. Brooks, David. "A Future in Human Capital." New York Times 13 Nov. 2005. Johnson, Dave. "Brazil Stops River Oil Spill Far From Key City." CNN. 19 June 2000. Reuters. 28 Apr. 2006 <www.cnn.com> Meyers, S., E. Mills, and A. Chen. "Building Data Visualization for Diagnostics, Operator Feedback, and Performance Optimization." ASHRAE Journal Dec. (1995): 63-73. Soltani, Babak. Finding Design Flaws in the Ballast Control System on the Pride do Rio Oil Platform. Undergraduate Thesis. University of Virginia Systems and Information Engineering. 2005. Vicente, Kim J. Cognitive Work Analysis: Towards Safe, Productive, and Healthy Computer-Based Work. Mahwah, NJ: Lawrence Erlbaum, 1999.
36
37
APPENDIX A: FULL SCREENSHOT OF TREEMAPS DATA VISUALIZATION SOFTWARE
