Testing the reliability, validity and usability of the prototype Anaesthetic Non-Technical Skills (ANTS-AP) behaviour rating system

Human error is the leading cause of critical incidents in the operating theatre. One approach to reducing human error has been to develop role specific taxonomies of Non-Technical Skills (NTS) which have been used for training, understanding performance and research. Examples include systems for anaesthetists (ANTS) , surgeons (NOTSS) and scrub practitioners (SPLINTS) , but not yet for the Anaesthetic Nurses and ODPs who assist anaesthetists in the operating theatre. A task analysis was performed to identify relevant NTS using interviews with anaesthetic assistants and anaesthetists, as well as a review of the Australian Incident Monitoring System database for the contribution of NTS to critical incidents. The system was then developed with focus group interviews and a subject matter expert Delphi questionnaire. This study aimed to assess the reliability, validity and usability of the prototype ANTS-AP system.

Rutherford, John1,2; Flin, Rhona2; Hellaby, Mark3; Caldwell, David1

1Dumfries & Galloway Royal Infirmary; 2Aberdeen University; 3North West Simulation Education Network

DISCUSSIONThe ANTS-AP system was generally received enthusiastically. The training the participants received was very short and some of the candidates had limited awareness or appreciation of human factors and non-technical skills, so the modest scores for reliability are not unexpected. The detailed results will be written up in a future publication. The system will need further investigation if it is to be used for assessment.

ACKNOWLEDGEMENTSScottish Clinical Simulation Centre for assistance in producing the videos used in this study.North West Simulation Education Network for access to their Human Factors on-line learning module.

REFERENCES1. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marking system. British Journal of Anaesthesia 2003; 90: 580-588.2. Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behaviour rating system. World Journal of Surgery 2008; 32: 548-556.3. Mitchell L, Flin R, Yule S, Mitchell J, Coutts K, Youngson G. Evaluation of the Scrub Practitioner List of Intra-operative Non-Technical Skills (SPLINTS) system. International Journal of Nursing Studies 2012; 49: 201-211.

RESULTS80 individuals expressed an interest in the study, and 58 attended workshops, with 48 available for analysis (8 forms incomplete, 1 incorrectly scored, 1 too inexperienced). There was good internal consistency for the elements with the categories (Cronbach’s alpha), but only weak to fair inter-rater reliability (ICC), test retest reliability (ICC) and accuracy (weighted kappa) compared to the authors rating. The exploratory factor analysis suggested that the items all formed one construct. The system was generally considered complete, and straightforward to use.

METHODSEthical approval was gained from Aberdeen University (PEC 0108131886), and confirmation of exemption by West of Scotland NHS Research Ethics Service. Simulated theatre cases were filmed at the Scottish Clinical Simulation Centre to illustrate differing examples of NTS (Situation awareness (SA), Team working (TW) and Task management (TM)). At least 45 Anaesthetic Nurses and ODPs with at least one year’s experience working in NHS hospitals were to be recruited for 80% power with 5% significance. They were given access to an on-line introduction to human factors to increase their awareness of human factors prior to attending a half day workshop. A separate video clip was used for orientation to the ANTS-AP system, and then they individually scored 12 video clips. After watching the videos, the participants were asked to rate the ease of use of the system, its completeness and suggest any potential alterations. Statistical analysis was performed using SPSS(v22).

John Rutherford, Dumfries & Galloway Royal Infirmary, Dumfries, DG1 4AP. Tel +44 1387 246246. E-mail: john.rutherford:@nhs.net

Prototype ANTS-AP System

Accuracy (Weighted κ)Test retest reliability (ICC)

Category Category Element Element Feedback on rating* rating* performance and debriefing notes Situation Gathering Awareness Information Recognising & Understanding Anticipating Teamwork & Co-ordinating Communication with team Supporting colleagues Asserting Task Planning & Management Preparing Prioritising and Problem solving Coping with pressure

SA

0.1 0.10.0 0.0

0.6 0.6

0.2 0.2

0.7 0.7

0.3 0.3

0.8 0.8

0.4 0.4

0.9 0.91.0 1.0

0.5 0.5

TMR S PSG CTW PA AS CP 0 10 20 30

ElementsCategories

40 50

Inter rater reliability (ICC)Internal Consistency (Cronbach’s α)

SA ElementsTW ElementsTM Elements

1 SA

0.1 0.10.0

0.6 0.6

0.2 0.2

0.7 0.7

0.3 0.3

0.8 0.8

0.4 0.4

0.9 0.91.0 1.0

0.0

0.5 0.5

9 TM3 R7 S11 PS2 G6 C5 TW10 P4 A8 AS12 CPVideos

Categories/Elements Participants

Categories/Elements

α

ICC

ICC κ