CS5032 Lecture 9: Learning from failure 1

LEARNING FROM FAILURE 1DR JOHN ROOKSBY

IN THIS LECTURE

Today - two lectures looking at how organisations can investigate and learn from failure

• Incident reporting

• Accident investigations

Incident reporting:

• It is important to learn from incidents irrespectively of whether they caused harm

• Aviation, Healthcare and the Nuclear industry routinely collect incident data, but with varying success

• Collecting incident data can be problematic – mainly because people don’t want to be blamed for highlighting incidents

Accidents are the “tip of an iceberg”• Incidents or “adverse events”

are far more common, and underlie accidents.

Incident reports highlight faults or report errors and near misses where there was potential for an accident.

If incidents are not reported, they may never be noticed by anyone other than those involved

What constitutes an incident and whether it warrants reporting can be a judgement

http://en.wikipedia.org/wiki/File:Iceberg.jpg

INCIDENT REPORTING SCHEMES

Incident Reporting Schemes are mechanisms for learning from errors and failures

Used in many safety critical domains:

• Healthcare, Aviation, Marine, Railways, Nuclear Power, Oil and Chemical Production

Increasingly used in complex, business critical environments:

• For example Data Centres

EXAMPLE REPORT - AVIATION

On pre-flight check I loaded the Flight Management Computer (FMC),

with longitude WEST instead of EAST. Somehow the FMC accepted it

(it should have refused it three times). During taxi I noticed that

something was wrong, as I could not see the initial route and runway on

the navigation map display, but I got distracted by ATC. After we were

airborne, the senior cabin attendant came to the flight deck to tell us the

cabin monitor (which shows the route on a screen to passengers)

showed us in the Canaries instead of the Western Mediterranean! We

continued the flight on raw data only to find out that the Heading was

wrong by about 30-40 degrees. With a ceiling of 1,000 ft at our

destination I could not wait to be on 'terra firma'. Now I always check

the Latitude/Longitude three times on initialization!”

EXAMPLE REPORT – PHARMACY Date of report: March 6th Reporter: Betty Jones

Date and time of incident: March 6th, 4.30pm

Description of incident: Warning message ignored by doctor. A PEP

(Post-exposure prophylaxis) medication kit was ordered by a doctor in the

emergency department. Standard PEPs contain lopinavir, ritonavir,

zidovudine and lamivudine. The patient was already using several

medications including venlafaxine, amitriptyline, bupropion, and fentanyl.

If this standard PEP had been administered to the patient there may have

been a harmful interaction between the ritonavir and the fentanyl.

When the doctor ordered the PEP, the IT system flagged up a warning

message saying there was a potential drug interaction problem between

ritonavir and fentanyl. This warning was ignored by the doctor, who later

explained to me that she didn’t read it because she was in a rush.

This interaction is potentially fatal, and I rejected the prescription when I

saw it. An alternative PEP kit was dispensed.


Reporting incidents, not just accidents, enables organisations to:

1. Identify why errors and failures occurred.

2. Identify why accidents DON’T occur. • What are the barriers that stop errors or failures escalating to

accidents?

3. Produce reminders of known hazards and workarounds and generally keep people thinking about safety and improvement

4. Share success stories

5. Allow information to be shared between sites, and (sometimes) between organisations

6. Produce adequate quantities of data for understanding general issues (human factors, regulatory weakness etc) or rare issues.


Lessons can be drawn from incident reports on an individual and collective basis:

• Individually:

• Reports are treated as a “war story”. This way the individual report is recounted in a meeting and discussed or can be posted in a newsletter.

• Discussion and learning takes place among practitioners• Collectively:

• Reports are collected together and can be analysed to identify themes and patterns

• Lessons have to be drawn out from a manager or specialist investigator.

REPORTING RATES

There is an inverse relationship between the number of incidents reported and the number of accidents

The number of incident reports is not a measure of incidents

• If an organisation has no incident reports, this does not mean it has had no incidents.

• If an organisation has many incident reports this does not mean that it has had many incidents.

Question: Over a twelve month period, organisation A has 0 incident reports; organisation B has 100; organisation C has 1000. Which is the safest organisation?

REPORTING RATESIncident reports in 1997 2000

Commercial Aviation: 22,908 26,623

General Aviation: 8,384 8,501

Accident rates per 100,000 flight hours

Commercial Aviation: 1.6 3.24

General Aviation: 7.19 6.3Accidents

Commercial Aviation: 147 (21) 148 (26)

General Aviation: 1845 (350) 1837 (344)

FACTORS INFLUENCING REPORTING RATES

Professionalism:• Reflection is an essential aspect of professional practice• Professional bodies encourage the accumulation of

knowledge and create circumstances in which this can be achieved

• In many cases incident reporting schemes arose from professional groups rather than within individual organisations


Trust and Blame • If someone feels they might be blamed for a report they are

less likely to write one• Blame may be from other practitioners, managers, the

media/public• It can also be socially problematic to write reports that

concern actions by other people• Organisations with a “blame culture” therefore have

problems in learning from failure• Enabling anonymous reporting can avoid issues of blame to

a certain degree, but successful schemes thrive when there is a “culture of trust”.


Design: • The design of a scheme also influences its success• How easy is it to report?

• Incidents are best reported when they are fresh in your mind, and reports are more likely to be completed if it is relatively quick and easy to do so.

• Paper vs electronic forms. Complex vs simple forms.• Feedback loops are also very important. People are more

likely to report if they see value in writing reports.


Why do commercial pilots/crew report more?

1. There is a ‘no blame’ environment. Although reports are screened for serious offences, the orientation is to looking for the root causes of error rather than blaming individuals.

2. There is more education about the value of reporting incidents

3. There is a pro-reporting culture, pilots are reprimanded for having too few reports

4. Commercial pilots have more to lose if they fail to report an incident. They are more likely to report, especially if they think someone else may have seen the incident

5. There is a workable separation between accident reporting and incident reporting. The media focus more on accident reports.


In the NHS (England) incident reporting in anaesthesiology fell massively when hospital wide reporting schemes were launched in mid 2000s to replace departmental schemes

• No atmosphere of trust across hospitals • Complex and cumbersome forms, or direct computer entry• No visible feedback loops• Led to parallel reporting schemes (in some cases reporting

was done in secret)

The situation improved over time

ANONYMITY AND REPORTING

The Paradox of anonymity: People often happier to report anonymously but the reports are of less value to an investigation

There are different ways to handle anonymity in incident reporting

• Open Schemes: Full disclosure of identity of reporter and those involved

• Confidential: Disclosure of identities to trusted third parties

• Anonymous: No disclosure of identity

Designing an anonymous system is very difficult. Anonymity is not a good substitute for trust

• Even if it is not clear who reported, it may be obvious who featured in the events described

• People might also be seen reporting, even if what they report is not clear.

GENERIC REPORTING SEQUENCE

Submit Report

Assess Report

Corrective Action

Publish Report and Corrective Action

SIMPLE, SMALL SCALE SCHEME

A report is submitted to report coordinator as soon as possible after an error

Report coordinator asks secretary to type up report

Reports are collated and sent to practitioners monthly

Reports are discussed in monthly meeting and corrective actions decided

LARGE SCALE SCHEMEContributor submits report

Third party validates and supplements report.

Submits to management and regulator

Management and regulator decide on corrective actions

Incident summary and corrective action published

in bulletin

Bank of previous reports

examined for similar incidents

if necessary

Regional and national

investigators contacted if necessary

Specialists (human factors, systems, etc) contacted if necessary

SMALL SCALE SCHEMES

Many incident reporting schemes begin as local, small scale schemes. For example a scheme might be initiated in one department in an organisation, or among a local professional group.

• Often higher levels of trust• The analysis can more readily draw from contextual knowledge• Focus on quick fixes (“make do and mend” culture)• Can be quick to react

LARGE SCALE SCHEMES

Large, Organisation-Wide and National Schemes

• More reports and greater coverage, but with inconsistencies between reports

• More opportunities to look for root causes• Greater overheads in analysing reports because of lack of

contextual knowledge• Trust much harder to maintain

International Schemes

• Some attempts at this, particularly within European Union. Tend to focus on bulletins and announcements

OPERATION ORCADIANAround ten years ago a boy died during an operation. His anaesthetic breathing circuit was blocked by a small plastic object.

• An initial assumption of the police was that this was a deliberate act of sabotage

Incident data was drawn together from across the NHS and a number of incidents were found where small plastic objects were blocking the breathing circuit

An investigation discovered the likely cause was that plastic caps from medications were sometimes finding their way into breathing tubes during storage

Changes were made to the ways in which breathing equipment is stored

Training was updated

And guidance was issued on how to spot when the breathing circuit is obstructed

MULTIPLE REPORTING SCHEMES

A problem other than a lack of reporting, is that some industries have multiple reporting schemes

• Different purposes and audiences• Different jurisdictions and authorities• Different geographic areas• Different approaches to confidentiality


UK Nuclear Power Industry

• NUPER (Nuclear Plant Event Reporting): Internal, private database of incidents in UK power industry

• UK HSE (Health and Safety Executive): Publishes full incident reports, and summary versions

• MHIDAS (Major hazards incidents database service): A bibliographic resource, maintained by Safety and Reliability Directorate

• INIS (International Nuclear Information System): Coordinated by international atomic agency in Austria

• PDR (Public Document Room): USA based resource, links to some incident reports and bulletins


Aviation - Three schemes is the UK

• CAA Mandatory Reporting System, and Voluntary Reporting System

• CHIRPS (Confidential Human Factors Incident Reporting Programme Scheme)

• AAIB Air Accident Investigations Board (UK) produces monthly bulletins

International Schemes: Civil Aviation Organisation operate the ADREP Accident/Incident Reporting System.

Europe: European Commission is trying to overcome report scheme compatibility issues with ECC-AIRS The European Co-ordination centre for Aircraft Accident Reporting

KEY POINTS

Incident reporting schemes are important to safety. Reports can focus on errors and near misses as well as failures

Trust is important for a successful schemes, people should not be blamed for reporting an incident

Reporting schemes need to be well designed if they are to be effective

Reporting rates do not correlate with incident rates. No reports does not mean there were no incidents.

Small scale schemes often focus on quick fixes rather than root causes. Large scale schemes are slower and more bureaucratic but can be more thorough.

FURTHER READINGC.W. Johnson, Failure in Safety-Critical Systems: A Handbook of Accident and Incident Reporting, University of Glasgow Press, Glasgow, Scotland, October 2003.

Full text online: http://www.dcs.gla.ac.uk/~johnson/book/

EXERCISE

1. What is the value of the following reports?

2. Should the people involved be blamed for these incidents?

3. Can you categorise events in these incidents using the GEMS slips-lapse-mistake model?

4. What lessons are drawn from these incidents?

5. Do the lessons learned from these incidents address the root causes of the problems? If not, why?

On pre-flight check I loaded the Flight Management Computer (FMC),

with longitude WEST instead of EAST. Somehow the FMC accepted it

(it should have refused it three times). During taxi I noticed that

something was wrong, as I could not see the initial route and runway on

the navigation map display, but I got distracted by ATC. After we were

airborne, the senior cabin attendant came to the flight deck to tell us the

cabin monitor (which shows the route on a screen to passengers)

showed us in the Canaries instead of the Western Mediterranean! We

continued the flight on raw data only to find out that the Heading was

wrong by about 30-40 degrees. With a ceiling of 1,000 ft at our

destination I could not wait to be on 'terra firma'. Now I always check

the Latitude/Longitude three times on initialization!”

A

Date of report: March 6th Reporter: Betty Jones

Date and time of incident: March 6th, 4.30pm

Description of incident: Warning message ignored by doctor. A PEP (Post-

exposure prophylaxis) medication kit was ordered by a doctor in the

emergency department to reduce the risk of HIV infection to a patient who had

been assaulted. Standard PEPs contain lopinavir, ritonavir, zidovudine and

lamivudine. The patient was already using several medications including

venlafaxine, amitriptyline, bupropion, and fentanyl. If this standard PEP had

been administered to the patient there may have been a harmful interaction

between the ritonavir and the fentanyl.

When the doctor ordered the PEP, the IT system flagged up a warning

message saying there was a potential drug interaction problem between

ritonavir and fentanyl. This warning was ignored by the doctor, who later

explained to me that she didn’t read it because she was in a rush.

This interaction is potentially fatal, and I rejected the prescription when I saw

it. An alternative PEP kit was dispensed.

B

Outcome: The pharmacists agree they must be vigilant when drug interactions have been overridden by doctors. The pharmacists recognise that doctors in the emergency department are very busy and that the system does not always meaningfully describe the seriousness of particular drug interactions.

B

Reporter: Anon Patient Sex: Male

ASA: 2: Relevant systemic disease

Urgency: 1: Routine; on distributed list

Factors: anaesthetist, organisational

The incident caused: 3: Transient abnormality with full recovery

How preventable do you think the incident would be by further resource? 1: Probably within current resource

What happened? The patient was for direct pharyngoscopy, a short but stimulating procedure so the plan was to use boluses of alfentanil and mivacurium. Both these drugs were in correctly labelled 10 ml syringes. Inadvertently I gave the mivavurium prior to induction instead of alfentanil. I did not realise my error for a few minutes. The patient initially appeared drowsy but agitated, breathing became shallow and saturation dropped to 85%. He developed multple VEs. On realising my error some propofol was given, the trachea intubated and over a short period of time his saturation and ECG returned to normal. We continued with the procedure. On recovery he had recall of what had happened and was quite distressed by it.

C

Lessons learned:

1. Correctly labelling syringes isn’t enough, especially when the colour of the labels is very similar. In this case both the labels that come with the drug are white. We use other visual aids first, syringe size probably being the most important.

2. Avoid drawing up muscle relaxants and induction agents in similar size syringes at the same time as other drugs, ie sux and fentanyl, thiopentone and augmentin.

3. In this case the part the cause for the error was that I was using a number of drugs that I dont usually use - thats when you should be extra vigilant.

C

Reporter: Anon Patient Sex:

ASA: 1: Fit

Urgency: 1: Routine; on distributed list

Factors: Equipment

The incident caused: 2: Transient abnormality unnoticed by patient

How preventable do you think the incident would be by further resource? 5: Not obviously by any change of practice

What happened? Patient was having a rigid bronchoscopy followed by submandibular gland excision. The patient was ventilated using a Sanders injector for the bronchoscopy, connected to the high pressure oxygen outlet on the anaesthetic machine (Blease Frontoline). Following the bronchoscopy, the patient was reintubated and conventionally for the next procedure. However, when the Sanders injector was disconnected from the oxygen outlet, the outlet valve jammed open, causing a massive leak of oxygen, enough to cause a complete failure of the anaesthetic gas supply to the patient. Fortunately, my initial response of fiddling with the leaking valve led to it closing and restoring normal function. If it had not closed, or another anaesthetist had reacted differently, the patient would have remained unventilated until an alternative system of ventilation could be obtained. Cylinder and piped medical air on the machine were of no value as all the gas supply was leaking out.

D

Lessons Learned: However good and reliable modern anaesthetic machines are, catastrophic oxygen failure can always occur – even bypassing the normal backup of cylinder supply, or medical air supply as in this case. The new Association of Anaesthetists machine checklist recommends checking that an alternative means of ventilating a patient is available and checked – this incident is a good reminder of how important that can be. I intend to use this incident as a teaching scenario from now on.

D

EDINBURGH G-JECI INCIDENT

Education

CS5032 Lecture 9: Learning from failure 1