Guide to Clinical Audit

A Guide to Clinical Audit

Dr. CHANDY ABRAHAM.Chief of Med. Staff.Baptist Hospital. B’lore.MS. DNB. MRCS. ADHA.

How do we ensure a high quality of service?

How do we improve our service?

One way is to ask questions about how, where and when we deliver the service

This is the essence of Clinical Audit – at its simplest it is asking whether we do the right thing, at the right time, to the right patient

Clinical Audit isn’t really anything new. Early proponents of clinical audit include Florence Nightingale, who studied infection rates at Scutari Hospital during the Crimean War (1854) and Dr Ernest Codman, who monitored surgical outcomes at his Massachusetts Hospital (1914)

Clinical Audit is…………

Clinical Audit has many definitions, this is the one above is used by NICE and the Healthcare Commission

“a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change”

(Principles for Best Practice in Clinical Audit 2002)

Clinical Audit

“a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change”

From the definition we can extract two key points about Clinical Audit

1) That we need to set explicit criteria

2) That we need to set standards by which to systematically review care

And also a third concept – that audit should lead to change

Clinical Audit…….Criteria

What do we mean by criteria?

The criteria (singular criterion) are the things that we are measuring in our audit. A criterion can cover a single event e.g. “Did Staff sign the medical record in black ink?” or can cover a multitude of separate events within one process e.g. “Was the patient assessed and treated within 2 hours of arrival in clinic?”

The main thing is that the criteria used are objective, unambiguous and should generally be answerable either yes or no.

Lets have a look at some possible criteria

Clinical Audit…….Criteria

Is the patient seen in a timely manner? NOT an audit criterion – timely is ambiguous.

Was the patient satisfied with their treatment?NOT an audit criterion – satisfaction is not objective. Could very well be an important area to look at, but it is not strictly an audit.

Was the patient seen by a specialist within two weeks of referral?Possibly an audit criterion – but is referral absolutely unambiguous? – does it mean within 2 weeks of the date the referral letter was sent, date referral letter was received or date patient was put on the waiting list? And what exactly does two weeks mean?

Was the patient seen by a specialist within 14 working days of their GP appointment?YES – an audit criterion we can know the dates of both appointments and unambiguously measure the time between them.

Clinical Audit…….Standards

Once we have decided on our criteria we can define our standards. The standard is “how many of the cases that we look at do we expect to meet our audit criterion”.

We could set the standard at 100% e.g. All patients will be seen within 4 hours of arrival at A&E.

Or we might consider 100% to be unrealistic and so set the standard at a lower e.g. 95% of children will receive the MMR vaccination.

Or we could set more than one standard e.g. 50% of stroke patients should be SALT assessed within 24 hours and 100% within 72 hours.


But how do we decide what the standard should be? Fortunately there are a lot of sources to help us set

fair and evidence-based standard for our audits. These include:

o Nabh guidelineso NICE guidelineso the Royal Colleges and professional bodieso Scientific research paperso Local good practice

more on this follows


NABH GUIDELINES:

The NABH has policies, procedures and guidelines covering many subject areas. It is straight forward to develop an audit based on policy

For example let us take Discharge Policy.AAC15: Organisation defines the content of the discharge summary.

So the audit tool would beCriterion a) Is the DS in the disch.patient’s notes YES/NO Standard is 100% YESCriterion b) Does the DS have a clear diagnosis YES/NO Standard is 100% YESCriterion c) Does it contain info on urgent care YES/NO Standard is 100% YES


Scientific research

Of course, the standards suggested by all of the national bodies are based ultimately on the scientific evidence.

The National Library for Health website is an excellent portal to a vast array of information on evidence based healthcare.

You can find links to Bandolier, Medline, Clinical knowledge Summaries (formerly PRODIGY), and the Cochrane Library.

http://www.library.nhs.uk/Default.aspx


he National Institute for health and Clinical Excellence or NICE is an independent organisation responsible for providing national guidance on promoting good health and preventing and treating ill health (www.nice.org.uk).

NICE provide guidance both on treatment methods (known as clinical guidelines) and the use of specific drugs (known as technical appraisal documents). Often these guidelines will come complete with an audit tool to help you monitor your practice.

NICE guidance is highly evidence-based and so is regarded as a “Gold Standard” by external appraisal organisations such as Welsh Risk Pool and Healthcare Inspectorate Wales

http://www.nice.org.uk/


The Royal Colleges and other professional bodies have a great interest in promoting professional and service development through the use of audit.

It is worth looking at the web sites of the Royal Colleges and other professional bodies to see what audit tools they may have developed.

http://www.rcplondon.ac.uk/news/news.asp?PR_id=354


Local Good Practice

It may be that you wish to audit something that doesn’t fall into any of the other categories. The interest here is seeing whether a situation is static or changing.

So for example I could do an audit using the criterion

“a reply is to be sent to any written enquiries received by my office within 2 working days” and the standard “all written enquiries to be responded to in this way”

However it could be argued that since I have set both the criteria and the standard myself it may mean I am not working to any agreed best practice. Wherever possible try to base your audit on policy or national guidelines

Clinical Audit…….Summary

Remember audit is about two things

a) An unambiguous criterion – ideally one that can be answered yes or no

b) A standard – how many of the cases you examine is it reasonable to expect to meet the criterion? This standard should be set using the best available evidence.“Gold Standard” NABH,Scientific evid. NICE etc.“Silver Standard” State Policy based on registry etc.“Bronze Standard” Locally agreed standards

Ready? Lets do an audit

Audit example

My topic of interest:shiny sports cars

Audit example - Criteria

I have decided to audit my car’s fuel efficiency by measuring the miles I get per gallon of fuel

Objective audit?Yes, this can be measured by anyone in the same way

Unambiguous audit?Yes, as long as I give the results expressed as miles per gallon every one will know what I mean

Audit example - Standard

It is important to set a fair and evidence-based criterion and standard, so I do some research and look at several evidence sources

Evidence a) The Environmental Protection Agency, an impartial government department, suggests I should get 17.2 MPG

Evidence b) The Manufacturers, Lamborghini, suggest 18.1 MPG (under ideal conditions in Tuscany)

Evidence c) The Royal College of Sport Car Drivers, a respected professional group, suggest 17.5 MPG

I decide that 17 MPG is a fair figure for the criterion and that a fair standard would be that my car always meets this performancetarget

Audit example

Our audit is ready to go

We have the subject – car fuel economy We have the Criterion – The car must achieve at least

17 Miles per Gallon We have the Standard – The car must achieve this

figure 100% of the time. Finally we need to decide if this is a one–off

measurement or a regular audit. In this case lets do a monthly audit

Results

Date result standard met? May 07 17.5mpg yes June 07 17.9mpg yes July 07 17.1mpg yes Aug 07 On Holiday n/a Sept 07 16.8mpg no Oct 07 16.4mpg no Nov 07 15.9mpg no

Results

Clearly my car is falling below an acceptable standard and I need to take it in for a service. Note that nothing disastrous needs to have happened to my car. The benefit of audit is that I can pick up small problems before they become serious ones.

Congratulations, you’ve just done an audit!

How to choose an audit topic

This simple checklist may help you decide what to audit

A suitable subject for audit: is one with direct importance for patient care a large number of patients are involved a high cost is involved good-practice guidance has been issued by a

professional or governmental body there is significant clinical concern over patient health

and well-being unsatisfactory outcomes have been reported –either

incidents or complaints


What makes a good audit?

Clear nationally agreed criteria – such as auditing to NABH, NICE standards or national Royal College audits

Audits that involve multidisciplinary teams to get a complete picture of patient care

A realistic appraisal of how difficult it will be to complete the audit both in terms of time and in being able to access the data.


Think SMART !

S – have I defined my audit topic well enough for it to be truly Specific

M – have I chosen criteria such that they are practical and Measurable

A – Have I enough resources/access to make the audit Achievable

R – in appraising the difficulty in completing the audit have I been Realistic

T – Can I give the audit enough effort to complete it in a Timely way.

OK, lets have a further example

Scenario : You wish to undertake an audit around the supply of insulin to diabetic patients who present with a doctors prescription.

Is this a good audit subject? We can decide by going down our checklist –remember it’s a guide, not a rulebook, we don’t have to answer ever question yes, but if you answer every question no, is it really a worthwhile audit?

Insulin supply – a good audit?

Important for patient health? Yes, Control of diabetes by the taking of insulin or oral hypoglycaemic agents has major

health benefits.

Large number of patients involved? Yes, Diabetes is a fairly common condition

High Cost? Not specifically relating to treatment but costs of error or non-compliance can be high

Good practice guidance available? Yes, this area is well researched

Is there concern over incidents or complaints? No, none at the moment

Is it multidisciplinary? Yes, involves more than one specialty, pharmacists and patients

Can the data be obtained easily? Yes, by discussion with patients and examination of prescriptions

So overall we can see this will be a worthy audit subject

Let us begin by setting measurable, unambiguous criteria

Does the prescription contain instructions as to dose and frequency?

Does the patient know how to take the medication? Is the patient compliant? Does the patient suffer side effects? Does the patient know their last blood glucose level? Does the patient smoke? Has the dose been added to the medication label?

There could well be others you can think of, but these were the ones actually chosen for this particular audit which was conducted at a NHS Trust in Wales in 2005

Let us then set our standards

1) Does the prescription contain instructions as to dose and frequency?

We feel this is important so the standard should be 90% YES

2) Does the patient know how to take the medication? We feel this is very important so the standard should be

100% YES

3) Is the patient compliant? We feel this is important but the standard should be more

that 85% YES since we cannot enforce total compliance.

4) Does the patient suffer side effects? We recognise that some people may suffer side effects,

however the evidence suggests it should be less than 15% of patients, so the standard should be less than 15% YES

Let us then set our standards

5) Does the patient know their last blood glucose level? We feel this is important so the standard should be

100% YES

6) Does the patient smoke? The evidence available to us suggests that smoking is

very harmful to diabetics therefore we feel the standard should be that no one smokes i.e. 0% YES

7) Has the dose been added to the medication label? We feel this is important so the standard should be

100% YES

Create your audit tool

Yes No total % of total that was yes

Was there Appropriate instructions?

Did the Patient know how to take the medication

Was the patient compliant?

Did the patient suffer side effects

Did the patient know last Blood Glucose?

Does the patient Smoke

Was the Dose added to label?

Collect your data

Yes No total % of total that was yes


36 patients 12 patients 48 patients 75%Did the Patient know how to take the medication

44 patients 4 patients 48 patients 91%Was the patient compliant? 42 patients 6 patients 48 patients 87%Did the patient suffer side effects? 7 patients 41 patients 48 patients 14%Did the patient know last Blood Glucose? 26 patients 22 patients 48 patients 54%Does the patient Smoke? 13 patients 35 patients 48 patients 27%Was the Dose added to label? 0 patients 48 patients 48 patients 0%

Be critical as you go along

Remember that you are using an audit tool. Like any tool it must do the job you want it to. If you find that the audit isn’t capturing the data you want or there is some unexpected difficulty don’t be afraid to stop and rethink the tool.

For this reason it can be a good idea to have a trial run before doing any large scale audits.

In this case the audit staff realise there actually two distinct patient group – those who are prescribed insulin and those who are prescribed oral hypoglycaemic agents. So they modified the audit tool to capture the data slightly differently – presented this way we see that the two patient groups have markedly different results for 4 of the 7 criteria

New audit tool with data for insulin dependent patients and those taking oral hypoglycaemic agents (HGA) recorded separately

Insulin YES

Insulin NO

Insulin % YES

HGA YES

HGA NO

HGA % YES


0 patients 8 patients 0% 36 patients

4 patients 90%

Did the Patient know how to take medication


4 patients 90%



6 patients 85%

Did the patient suffer side effects?

2 patients 6 patients 25% 5 patients 35 patients

12.5%

know last Blood Glucose?

3 patients 5 patients 37.5% 23 patients

17 patients

57.5%

Does the patient Smoke?


28 patients

30%



0%

How to write up your audit

Begin with an introduction giving a brief background as to why you are doing this audit and, if you are using external criteria and standards eg NICE guidance, what these may be.

Generally you will want to give enough detail about what you did to allow someone else to repeat your audit just from reading your report

Don’t forget to date your report and to put your name or job title – someone might want your advice if they are doing a similar audit.

How to write up your audit

Title: Insulin and HGA supply

Background: Good medication control is essential in the treatment of diabetes Criteria and Standards used: Audit was based on that of the Pharmaceutical Society of Finland April 2006Sample: All diabetic patients presenting at the pharmacy with prescriptions in the month of September 2006. 48 patients took part in the auditReport date: January 2007Author: J G Pharmacist esq.

But how do I present the data?

Do I need to do a complex statistical analysis?

Do I need to do a complex statistical analysis?

Generally no, unlike research, most audits will not require heavy number crunching

In fact, a simple graphical display is often the most effective method of sharing your data. Don’t be tempted to overcomplicate things just because your computer will let you!

Lets have a look at those diabetic patients in the audit we just did who are smokers

Presenting the data -Diabetics who smoke

SmokerNon-smokers

S10

5

10

15

20

25

30

35

This is typical of the many graphs that Excel and other programscan do for you. Very stylish- but also very difficult to understand the eye is drawn all over the graph and finds it hard to make sense of it

Presenting the data -Diabetics who smoke

Smoker27%

Non-smokers73%

This is exactly the same data but shown in a different way. The readercan quickly grasp that just over a quarter of our diabetic patients aresmokers

Of course you will then want to include your audit data

Insulin YES

Insulin NO

Insulin % YES

HGA YES

HGA NO

HGA % YES



4 patients 90%

Did the Patient know how to take medication


4 patients 90%



6 patients 85%

Did the patient suffer side effects?


12.5%

know last Blood Glucose?

3 patients 5 patients 37.5% 23 patients

17 patients

57.5%

Does the patient Smoke?


28 patients

30%



0%

But also consider an easy to read summary

Patients taking insulin Patients taking HGA’s

criteria Pass or Fail Pass or FailAppropriate instructions Fail PassPatient knowledge Pass Fail

patient compliance Pass Failpatient side effects Fail Passlast Blood Glucose known Fail Failpatient Smoking Fail FailDose added to label Fail Fail

Discussing the data

A brief discussion will help flesh out the bare bones of the audit perhaps take each point in turn

Criterion 1 It was notable that whilst most patients taking the tablets had appropriate documentation with the prescription, none of those taking insulin did

Criterion 2 On the other hand, whilst insulin taking patients were very knowledgeable about their treatment, tablet-taking patients were much less sure of what to do.

Criterion 3 Again, whilst insulin-taking patients were very compliant with their treatment, tablet-taking patients were much less so.

Criterion 4 Whilst overall side effects were within acceptable levels when look at as separate patient groups insulin-taking patients may be suffering an unacceptable level of side effects

Criterion 5 Both patient groups fall below the standard we set for knowing their last blood glucose

Criterion 6 A higher than desirable number of patients smoke Criterion 7 – A disaster apparently – no labelling at all!

Be critical after you’re done

Why did all the patients taking insulin not have detailed prescriptions? To complete your audit maybe some follow up questions are needed. Ask the doctors who prescribe whether there are good reasons for this or not.

It might be that all the patients involved are long-standing insulin takers who are no longer thought to require detailed prescriptions –but does this mean the procedures have become casual, even slack as a result? Would a new insulin taking patients also not get a detailed prescription?


Why did some the patients taking tablets not have a good understanding of how to take their medication?

Is it perhaps because tablets are thought to be less of a problem than injections and therefore less time is devoted to explaining to patients how the drugs should be taken? Do older patients find it difficult to open packaging and bottles?


Can you cross-check your data? Did the 6 patients who were non-compliant include the 4 patients who did not have a good understanding of how to take their medication? Otherwise you might mistakenly think compliance is the issue when in fact it is due to a lack of patent understanding


Is your data hiding something? When looking at side effects the overall rate was an acceptable 14%. However when broken down into sub-groups we see that the whilst the tablet-taking patients had an acceptable 12.5% side effect rate, the insulin taking patients had an unacceptably high side effect rate of 25% and would merit review.


On the face of it only 54% of patients know their last blood glucose levels, a figure that gives us concern.But talking to audit staff reveals that we have asked a rather weak question. Remember one of our rules for good audit criteria was that they should be unambiguous.

What exactly are we asking here:

A. That patients know their last blood glucose level and remember it to two decimal places when asked?

B. Patients did know their last blood glucose level and approximately remember what it was when asked?

C. Patients do regularly check their blood glucose levels and know the appropriate action to take but can’t quite remember what it was last time?


Compare the question on glucose levels to the next question on smoking. We can be confident that this criterion is more robust in that patients will absolutely remember if they smoke or not! However we must be aware that some patients might be reluctant to admit to doing something that is known to be bad for them!


Finally why did we score 0% for dose on label are staff unaware of procedure? are staff aware but unwilling to adopt new practice? has new legislation just implemented?

However if you suspect a particularly poor outcome for a given criterion you should explain to staff why you are measuring it. It is often the system that is at fault not the individual


Perhaps the reason for the 0% score was that it was a technical impossibility to score any better. Perhaps we know that we have no printers that can print on the label?So why measure something when you know the answer? a) Identify risksb) highlight concernc) force actiond) provide baseline data to measure future improvement

Formulate action plan

This in many ways is the most important part of the audit.

What changes would you like to see in place to address the issues that your audit has found? Your action plan might look something like this;

Criterion 1 Agree with doctors what details should be on all patients’ prescriptionsNew audit standard – all prescriptions to contain agreed details

Criterion 2 and 3 Ask staff to review what information is given to improve patients understanding of their medication.

Criterion 4 Ask the doctors to review the side effects suffered by insulin-taking patientsNew audit subject – specifically investigate insulin-taking diabetics in your area to see if there is generally a higher than expected rate of side effects

Formulate action plan

Criterion 5 Change the wording of the audit standard to be less ambiguous

Criterion 6 Ask all diabetic patients who present if they are smokers.New audit standard – all diabetic smokers to be given advice on smoking cessation

Criterion 7 Put in a request for funding to obtain printers capable of printing labels.

The next steps

Audit gives a formal record of what is going on.

Audit highlights current strengths as well as possible weaknesses.

Audit provides a benchmark against which we can measure change.

Audit gives us a mechanism to consider what we do and how we do it.

Audit is a process not an event – then end of one audit should provide the information that allows the planning of the next audit.

Audit is a spiral that strives to push standards ever higher.

Audit is part of the process not the end point

Clinical audit:All patients must take

2 red pills per week

Patient experience:Did the patient recoveras well as they expected?

Clinical effectiveness: Did the patients recoveras well as we expected?

Patient satisfaction: Are the pills easy to take?

Patient interface: Do patients find it easy to take the correct dose with blue pills or red pills?

Research: Do patients

recover better if they take 2 red

pills or 3 blue pills per week?

How audit fits with the big picture

Documents

Guide to Clinical Audit