Dpc

HD

h

�����

a

ARRA

0h

Nuclear Engineering and Design 255 (2013) 212– 225

Contents lists available at SciVerse ScienceDirect

Nuclear Engineering and Design

j ourna l ho me pag e: www.elsev ier .com/ locate /nucengdes

evelopment of a quantitative evaluation method for non-technical skillsreparedness of operation teams in nuclear power plants to deal with emergencyonditions

o Bin Yim, Ar Ryum Kim, Poong Hyun Seong ∗

epartment of Nuclear and Quantum Engineering, Korea Advanced Institute of Science and Technology, 373-1, Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea

i g h l i g h t s

We selected important non-technical skills for emergency conditions in NPPs.We proposed an evaluation method for the selected non-technical skills.We conducted two sets of training, 9 experiments each, with real plant operators.Teams showed consistent non-technical skills preparedness with changing scenarios.Non-technical skills preparedness gives plausible explanations why teams fail tasks.

r t i c l e i n f o

rticle history:eceived 17 April 2012eceived in revised form 27 August 2012ccepted 1 September 2012

a b s t r a c t

Many statistical results from safety reports tell that human related errors are the dominant influenc-ing factor on the safe operation of power plants. Fortunately, training operators for the technical andnon-technical skills can prevent many types of human errors. In this study, four important non-technicalskills in safety critical industries – medical, aviation, and nuclear – were selected to describe behaviorsof operation teams in emergency conditions of nuclear power plants (NPPs): communication, leader-ship, situation awareness, and decision-making skills. Also, preparedness of the non-technical skills wasdefined, and a quantification method of those skills called NoT-SkiP (Non-Technical Skills Preparedness)was developed to represent ‘how well operation teams are prepared to deal with emergency condi-tions’ in the non-technical skills aspect by analyzing monitoring-control patterns and a verbal protocol.Two case studies were conducted to validate the method. The first case was applied to Loss of CoolantAccident (LOCA) and Steam Generator Tube Rupture (SGTR) training. Independent variables were sce-nario, training repetition, and members. Relative values of the NoT-SkiP showed a consistent trend withchanging scenarios. However, when training was repeated with the same scenario, NoT-SkiP values ofsome team were changed. It was supposed that leaders of some teams exerted their knowledge acquiredfrom the previous training and gave up thoroughness of using procedures. When members especiallywho play a dominant role in teams were changed, values of the NoT-SkiP were significantly changed.Nine teams participated in Interfacing System LOCA (ISLOCA) training; six teams failed to complete thegiven task, and three teams succeeded. The comparison of NoT-SkiP values of ISLOCA case with Operator

Performance Assessment System (OPAS), which mainly check operators’ knowledge and the technicalskills, developed by OECD Halden Reactor Project (HRP) gave a plausible reason why teams failed as wellas intuitive results on ‘what to be considered in the non-technical skills aspects of teams’ for the nexttraining. The proposed method requires more experiments in order for it to be established on a firmfoundation; however, it certainly gives one possible way to supplement existing training strategy in the nuclear industry.

∗ Corresponding author. Tel.: +82 42 350 3860; fax: +82 42 350 3810.E-mail address: phseong@kaist.ac.kr (P.H. Seong).

029-5493/$ – see front matter © 2012 Elsevier B.V. All rights reserved.ttp://dx.doi.org/10.1016/j.nucengdes.2012.09.027

© 2012 Elsevier B.V. All rights reserved.

1. Introduction

It has long been recognized that human reliability is one of

the most important determinants of system safety (Roth et al.,1994; Hollnagel, 2005). Many statistical results also tell that humanrelated errors are the dominant influencing factor, without doubt,on the safe operation of the plants (NEA, 2004). According to

ring an

d(rcpsfcfhiHh(ca

wtoadTatram

todsoAm

tttspemiTsmteD

2

2

ktih‘fme

critical industries are summarized in Table 1.The most popular training method for the non-technical skills

is the Crew Resource Management (CRM). The main purpose of theCRM is to reduce human errors and improve team performance

Table 1Non-technical skills in aviation, medical and nuclear industries (Crichton and Flin,2004; Sharma et al., 2011; Arora et al., 2011; O’Connor et al., 2008; Wang et al., 2010;Damme et al., 1998).

Industries Non-technical skills

Aviation Leadership, decision-making, situation awareness,workload management, team coordination

Medical Situation awareness, decision-making,communication and teamwork, leadership, stress

H.B. Yim et al. / Nuclear Enginee

ata recorded in the Operational Performance Information SystemOPIS; opis.kins.re.kr) in Korea, about 20% of accidents are solelyelated to human actions and this figure rises to about 50% whenounting events that have two or more causes. The human errorroblems have been viewed in many ways. One approach is to con-ider those problems as ‘active failures and passive failures’. Activeailures are unsafe acts committed by people who are in directontact with systems. They take a variety of forms: slips, lapses,umbles, mistakes, and procedural violations (Reason, 1990), andave been scrutinized by many researchers in a human reliabil-

ty analysis (HRA) field to quantify safety of NPPs. A recent trend ofRA is not only to put just human failures but also to consider moreuman like models to probabilistic safety assessment (PSA). Cepin2008a,b) identified dominant human failure events and signifi-ant parameters as inputs of PSA and modeled consecutive humanctions for more realistic results of PSA.

Passive failures are often caused under certain ‘latent conditions’hich lie dormant in both systems and operators or opera-

ion teams, including degree of independence among individuals,perator attitudes/biases, approach to implementing procedures,nd so on (Kolaczkowski et al., 2005). These conditions areirectly/indirectly connected to the non-technical skills of teams.hen, how can latent conditions in team characteristics be foundnd remedied? A recent analysis of Korean accident cases showedhat about 50% of them could be prevented from recurrence byehabilitation of operation and maintenance personnel. This kind ofnalysis result led to calls to develop human factor related trainingethods.But, so far, contents of training have been focused on improving

he technical skills of operators. Actually, the non-technical skillsf operation teams have been overlooked by the nuclear industryespite the fact that the importance of those skills has been empha-ized by regulatory bodies and many research institutes as a partf safety culture in NPPs (IAEA, INSAG-15, 2002; NEI 09-07, 2009).fter the Fukushima accident, the nuclear industry started to payore attention to safety cultural aspects of operation teams.The purpose of this study is to identify the non-technical skills

hat are crucial to the management of emergency conditions ando find quantitative methods to evaluate those skills. Four non-echnical skills were selected and an evaluation method for thoseelected skills was proposed. The proposed method provides a com-arison of ‘relatively weak and strong points of teams’ to deal withmergency conditions. It is clear that the further expansion of theethod to general conditions, which is to say not only emergency,

s necessary. The reminder of this paper is organized as follows.he general background of and necessity for this study are pre-ented in Section 2. Section 3 describes the development of theethod called Not-SkiP (Non-Technical Skills Preparedness). Sec-

ions 4 and 5 cover a feasibility of the method by conducting severalxperiments of emergency operation training in reference plants.iscussions related to the feasibility study and conclusions follow.

. Operation skills training

.1. Role of training

Training is defined as “the systematic development of thenowledge, skills and attitudes (KSAs) required by an individualo perform adequately a given task or job” (Armstrong, 1997). Itmplies that the role of training is a right mix of KSAs of trainees andelps jobholders to perform tasks successfully. Therefore, the term

performance’ is interwoven with training. In order to achieve per-ormance improvement, especially in the nuclear industry, training

ust lead operators to the enhancement of professional knowl-dge and skills both at individual and team levels. It should equip

d Design 255 (2013) 212– 225 213

operators to respond appropriately to emerging challenges like areactor tip or perturbations of plant parameters as well as appro-priate changes in attitudes.

2.2. Training in NPPs

2.2.1. General training strategy and concerns in KoreaMost countries set a limit to operating NPPs only by license

holders, and those licensed operators must be retrained (or reha-bilitated) within a certain period of time written in the law. InKorea, utilities give main control room (MCR) operators trainingof 114 h/yr to teach them basic knowledge and skills of given tasks,as well as changes in facilities, procedures, and regulations.

Among many training courses, simulation training is espe-cially taken to improve operators’ ability to cope with variousconditions. Abnormal conditions in NPPs are virtually uncount-able and most of them are unknown. For this reason, traininghas been more focused on simulated emergency conditions basedon design basis accidents (DBAs), which are thought to be wellunderstood by engineers and training designers. Trainees attainproficiency to diagnose perturbed conditions and mitigate thoseconditions through pre-designed training. However, this kind oftraining focuses on teaching particular situations or improving thetechnical skills and thus impedes operators’ ability to handle unex-pected situations.

2.2.2. Non-technical skills trainingIt has been well understood that the possibility of human errors

and inadequate team competency is high in large and complex pro-cess control industries. For this reason, training social and cognitiveskills has been an issue, particularly in safety critical industries.Social and cognitive skills are generally called the team skills or thenon-technical skills (non-technical skills for the rest), and they tendto be used interchangeably. The non-technical skills are definedas the cognitive and social skills of team members, not directlyrelated to control, system management, and standard operatingprocedures (Crichton and Flin, 2004). They encompass leadership,decision-making, situation awareness, workload management, andteam coordination, etc. (O’Connor et al., 2008; Wang et al., 2010).As mentioned, nuclear regulatory bodies recommend contractorsto have those skills as a basic requirement when hiring and promot-ing employees. Also in the aviation industry, especially, Europeanaviation legislation requires that the pilots in multi-crew cockpitsought to be trained and assessed in both the technical and non-technical skills (CAA, 2002). The medical industry has recently paidits attention to identifying the non-technical skills in surgical oper-ations and has developed tools to assess them (Sharma et al., 2011;Arora et al., 2011). The non-technical skills in three different safety

management, task managementNuclear Communication, teamwork, stress management,

decision-making, leadership, situation awareness,feedback, co-ordination

2 ring an

baae(tPtftretB1

2

aerCare

aesdmfiescss

3c

3

Trkaestto

3

iatcn

3

m

tors to cases, the ‘centrality’ can be used to represent the structureof operators’ communication. The physical meaning of the ‘central-ity’ is the relative degree of a specific node in the whole networkwhere degree means the total number of nodes that are directly

Table 2The speech act coding scheme for NPPs operation personnel.

Communication category Definition

Announcement A statement corresponding to thenotification to other operationpersonnel or the public, which givesinformation about something hashappened or will happen

Announcement Acknowledgment A statement clarifying the message of‘Announcement’ was received

Observation A statement describing the status of acomponent or system

Observation Acknowledgment A statement clarifying the message of‘Observation’ was received

Command A specific order to manipulate anobject

Command Acknowledgment A statement clarifying the message of‘Command’ was received

Reply An answer for ‘Inquiry’Reply Acknowledgment A statement clarifying the message of

‘Reply’ was receivedSuggestion A statement pertaining to the

recommendation of a specific action oridea

Suggestion Acknowledgment A statement clarifying the message of‘Suggestion’ was received

Judgment An expression pertaining to thejudgment of an on-going situation

Judgment Acknowledgment A statement clarifying the message of‘Judgment’ was received

Inquiry A statement for askingCall A call for a specific person for

communicationCall Acknowledgment A statement clarifying the message of

‘Call’ was received

14 H.B. Yim et al. / Nuclear Enginee

y enhancing the non-technical skills of teams. CRM experts inviation say that the CRM cannot be the perfect solution to man-ge human errors. Only, it enables the early detection of possiblerrors and focuses on minimizing damage caused by human errorsHelmerich et al., 1999). Yet, the CRM suggests reasonable solutionso develop training regimes (Cannon-Bowers et al., 1991; Salas andrince, 1999; Cacciabue, 2004; Salas and Kendall, 2004). Strangely,he nuclear power industry, which has a strong tradition of humanactors research in safety, seems to have paid rather less attentiono psychological or non-technical aspects of its teams’ emergencyesponse capabilities (Crichton and Flin, 2004). There are only fewxamples of applying the CRM into practice in the nuclear indus-ry: INPO recommendations, a CRM training program developed byritish Energy, and that by Korea Institute of Nuclear Safety (INPO,993; Belton, 2001; Kim et al., 2009).

.2.3. Emergency operating procedures trainingEmergency operating procedures (EOPs) are viewed as reference

nd inescapable aspects of safety in real situations as well as duringmergency condition training. They can be seen as the laws to beespected in accident situations. But, events like the TMI and thehernobyl accidents have shown that procedures alone were notn absolute and invariable guarantee of safety (Dien, 1998). As aesult, symptom-based EOPs were developed and widely used tonhance safety of NPPs.

As EOPs have taken a symptom-oriented approach or a hybridpproach, training also needs to change its approach to maximizefficacy of employing those procedures. Unfortunately, trainingtill takes the event-oriented approach, and there has been noirect application of the non-technical skills to EOPs training at theoment. It is highly possible that repeated training with a small and

nite set of scenarios may lead operators to stereotype or confinemergency conditions to only what they have been trained. Whenuch a mental situation happens, it blocks a diversity of operators’ognitive activities and may cause a biased diagnosis. So, it is neces-ary to supplement traditional ways of training with other aspects,uch as an appropriate usage of procedures.

. Non-technical skills preparedness for emergencyonditions in NPPs

.1. Purpose

The purpose of developing the method of evaluating the Non-echnical Skills Preparedness (NoT-SkiP for the rest) is to supporteasons why some teams fail and some succeed even thoughnowledge and the technical skills of all teams are assumed to bepproximately the same. The physical meaning of the NoT-SkiP formergency conditions is the team’s capability of the non-technicalkills to deal with emergency conditions using EOPs. Namely, it ishe evaluation method of ‘how well a team is prepared in the non-echnical skills aspect’ to manage any given emergency conditionsf NPPs.

.2. Data analysis methods

Five non-technical skills were investigated to be importantn safety critical industries: communication, leadership, situationwareness, decision-making, and teamwork skills. However, theeamwork skill was not dealt in this study because it was multi-oupled to other skills. Thus, the NoT-SkiP considers the rest fouron-technical skills.

.2.1. Verbal protocol analysisA verbal protocol analysis was performed to quantify com-

unication, leadership, and situation awareness skills of teams

d Design 255 (2013) 212– 225

under NPPs’ emergency conditions. Contents and the structure arethe most commonly considered characteristics for communica-tion analysis in the nuclear industry (Reinartz and Reinartz, 1992;Chung et al., 2004). For communication contents, verbal protocolanalysis based on the speech act coding scheme for NPPs operatorshas been proven to be effective (Kim et al., 2010, 2011), and theSocial Network Analysis (SNA) has been used for communicationstructure analysis in many areas (Ahuja and Carley, 1999; Yang andTang, 2004). The speech act coding scheme specialized for operatorsin NPPs and its brief descriptions are tabulated in Table 2.

The SNA is one of the methods to acquire meaningfulinformation from a given social group and has been adopted,besides communication structure analysis, in various fields, suchas psychology, education, sociology and so on (Scott, 2000;Henttonen, 2010). The purpose of the SNA is to provide insight-ful information and inferences on the organization and structuralproperties of a network, given its nodes and relations (Cioffi-Revilla and O’Brein, 2007). The process starts from creatingmatrix ‘A’, which represents a ‘cases-affiliations’ relationship.The matrix ‘A’ is a ‘two-dimensional’ relationship matrix thatconsists of two ‘one-dimensional’ relationships. Therefore, theone-dimensional relationship matrix, such as ‘cases-cases’ and‘affiliations-affiliations’ adjacent matrices can be produced eitherby A × AT or by AT × A matrix operations, where AT denotes thetranspose of matrix ‘A’. The SNA provides many measures to expressproperties of social networks, such as centrality, density, and so on.When applying communication contents to affiliations and opera-

Call Identification A receiver’s response corresponding tocaller’s self-identification

Call Identification Acknowledgment A statement clarifying the message of‘Call Identification’ was received

H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225 215

e mod

ca

C

wr

ot

3

qitaag

bot

Fig. 1. Two simple cognitiv

onnected to a particular node. The centrality can be also expresseds Eq. (1)

entrality =∑n

j=1xij(i /= j)

n − 1(1)

here n is the number of nodes and xij is the total number of directelations from the node i to another node j.

Roles of speakers in communication contents and steps thatperators discussed were also analyzed for deep understanding ofheir behavioral details.

.2.2. Monitoring-control patterns analysisThen, how can the ‘decision-making skill’ be assessed in a

uantitative and objective manner? Defining a cognitive models a fundamental step to build an assessment method of cogni-ive activities. There are some models using three basic humanctions: information gathering, decision-making, and controlctions (Sheridan, 1981; Chang and Mosleh, 2007). Schematic dia-rams of two cognitive models are compared in Fig. 1.

Monitoring is a basic activity of information searching and cane categorized by various ways. This study regards different typesf monitoring as different sources of inputs for a human cogni-ive activity. Fig. 2 shows input criteria of this study and their

Fig. 2. A Venn diagram of input criteria of information sources.

els of operation personnel.

definitions. As far as emergency conditions are concerned, thereare three types of monitoring as below (Roth et al., 1994).

Procedure-driven monitoring refers to monitoring that is deter-mined by procedures that include explicit directives to monitor aparameter: an area of P. Procedures have information that is notrelated to a specific scenario or event because symptom-orientedEOPs consider all the possible accidents in a diagnosis phase.

Data-driven monitoring refers to monitoring that is triggeredby salient external stimuli such as alarms: an area of (D(e) − P). Anarea of D(e) becomes as big as an area of T(e) as time goes andreaches to the end of a scenarios or an event.

Knowledge-driven monitoring refers to monitoring that isdriven by an internally generated perceived need for a pieceof information: an area of (U − (P∪D(e)). Information sources forknowledge driven monitoring are either in the area of T(e) whenteam’s situation awareness (SA for the rest) is correct or in an areaof E when team’s SA is incorrect.

Variations in quantity of the SA amount resulting in certaindecision-making are hardly described only by operators’ monitor-ing actions because monitoring itself is just an action of informationacquisition. But, if teams show different patterns of inputs and out-puts, i.e. information acquisition and the control action, for thesame situation, they each have their own ways of decision-making.

Based on monitoring types and the basic cognitive model previ-ously mentioned, monitoring-control patterns were defined to seedifference in the decision-making skill of teams. This study did notpay too much attention to details in the decision-making processbut considered cognitive activity cycles to clarify the skill to makea decision. By repeating inputs and outputs of a cognitive activ-ity cycle, SA of teams reach to make a decision, which is stronglyrelated to performance, either to a right direction or to a wrongdirection.

There were four premises to line monitoring-control patternsup in a cognitive activity demanding order.

Premise 1: According to the Rasmussen’s decision ladder modelshown in Fig. 3, the dotted arrows represent shortcuts to the exe-cution status (Rasmussen and Goodstein, 1985). If operators followprocedures and the situation they are confronting is consistent withthe procedures, then operators are taking shortcuts directly to thelast status of knowledge. This means that they do not need highercognitive activities in certain steps of procedures.

Premise 2: It is obvious that the control action or decision-making needs more cognitive activities than confirmation ofinformation to update one’s SA because the control action ordecision-making is a resultant of updated SA according to the model

216 H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225

s deci

o1

th

gnf

oaiitx

Fig. 3. Rasmussen’

f SA in dynamic decision-making by Endsley in Fig. 4 (Endsley,995).

Premise 3: When operators are alerted from alarms, they needo use cognitive resources whether the alarms fit to the SA theyave been forming or to the procedures they are conducting.

Premise 4: For knowledge-driven monitoring, operators areoing to the top of the decision ladder in Fig. 3 and use their cog-itive resources as much as possible to retrieve related knowledge

rom their long-term memory.If three types of monitoring above mentioned and two types

f outputs, i.e. information confirmation and the control action,re considered, there can be created 30 different types of mon-

toring patterns: (3P1 + 3P2 + 3P3) × 2. A reasonable assumptions necessary for a practical use of the proposed method. So,hree conditions were suggested to reduce markings on the-axis.

Fig. 4. Endsley’s model of SA in dynamic decision-m

sion ladder model.

Condition 1. Every procedural step is considered as a completeset of a monitoring-control pattern: either the monitoring-controlpattern 1 or 2. For example, knowledge or data-driven monitoring,which is in between procedural steps without information confir-mation, and ‘the control action or decision-making’, is not countedfor the monitoring-control patterns 3 or 6. If operators conducteda procedural step with these monitoring-control patterns, and itfits to their knowledge, operators will just proceed to the next stepbecause procedures are an abstract form of optimal paths based onknowledge.

Condition 2. There is no temporal order, and different numbers of

monitoring actions in each monitoring-control pattern are equallytreated. For example, both a set of four data-driven monitoringactions followed by two knowledge-driven monitoring actions anda set of a knowledge-driven monitoring action followed by two

aking and automaticity in cognitive process.

ring an

dm5

Cnanbkw

wM

M

M

M

M

M

ct1mm

3

bea

tt

3

tami

awatiIcwmaw‘eanb

q

H.B. Yim et al. / Nuclear Enginee

ata-driven monitoring actions with the control action or decision-aking are the same one cycle of the monitoring-control pattern

.

ondition 3. Confirmation of knowledge-driven monitoring isot considered since the term ‘knowledge-driven monitoring’lready involves confirmation of knowledge. When the operatorseed to confirm what they checked, which has been conductedy knowledge-driven monitoring, they will perform anothernowledge-driven monitoring. The knowledge-driven monitoringith the control action or decision-making is only valid.

Based on these premises and conditions, six monitoring patternsere defined.onitoring-control Pattern 1: procedure-driven monitoring → informationconfirmation (according to premises 1, 2 and conditions 1, 2).onitoring-control Pattern 2: procedure-driven monitoring → controls ordecision-making (according to premises 1, 2 and conditions 1, 2).onitoring-control Pattern 3: data-driven monitoring → informationconfirmation (according to premises 2, 3 and condition 2).onitoring-control Pattern 4: data-driven monitoring → controls ordecision-making (according to premises 2, 3 and condition 2).onitoring-control Pattern 5: Procedure or data-drivenmonitoring → knowledge-driven monitoring → controls ordecision-making (according to premises 2, 4 and conditions 2, 3).onitoring-control Pattern 6: knowledge-driven monitoring → controls ordecision-making (according to premises 2, 4 and condition 2, 3).

The number in each monitoring-control pattern is listed in theognitive activity demanding order, so the monitoring-control pat-ern 6 internally needs more cognitive activities than the pattern. In other words, the monitoring-control pattern 1 is cognitivelyore automatized than the pattern 6 based on the model of auto-aticity in cognitive processes in Fig. 4 (Endsley, 2000).

.3. NoT-SkiP under emergency operation in NPPs

The non-technical skills are often descriptive, and thus theiroundaries are sometimes not clear and such skills can be depend-nt on interpretation. Generally, there is neither a perfect way norn absolute function to represent human related skills.

To quantitatively measure four non-technical skills just men-ioned, some concepts were introduced based on early researcheso represent each skill. Detailed explanations are as below.

.3.1. Communication skillCommunication Completion. Integrity of contents is an essen-

ial element of communication. There is no doubt why industriesre enforcing operators to use a three ways communicationethod. Thus, the ‘communication completion’ concept was

ntroduced for the communication skill.Communication completion is measured by a verbal protocol

nalysis using ‘acknowledgement’ in the speech act coding schemeith the SNA to determine interpersonal communication integrity

mong many properties of communication. Acknowledgment inhe speech act coding scheme occurs when a receiver of a messagen communication has heard and understood what a sender said.n this sense, actions of acknowledgment can be a barometer ofommunication integrity. Centrality values of acknowledgementsere used to represent communication completion. Acknowledge-ents of ‘announcement’, ‘observation’, ‘command’, ‘suggestion’,

nd ‘judgment’ in the nuclear specific speech-act coding schemeere counted as well as ‘reply’. ‘Reply’ plays the same role as

acknowledgement’ for ‘inquiry’. ‘Reply-acknowledgement’ isxcluded because it is merely a double-counting of ‘reply’. ‘Call-cknowledgement’ and ‘call-identification-acknowledgement’ are

ot counted either because they are not directly related to SAuilding.

Euclidean norm with all acknowledgment values was used foruantifying communication completion, as shown in Eq. (2). By

d Design 255 (2013) 212– 225 217

doing so, the impact of ‘Sug. Ack’ and ‘Jud. Ack’, which are supposedto be less related to procedural conduction, on communicationcompletion can be reduced.

√(Ann. Ack)2 + (Obs. Ack)2 + (Com. Ack)2 + (Sug. Ack)2 + (Jud. Ack)2 + (Reply)2

(2)

3.3.2. Leadership skillSupportiveness of SA Building. Leadership is so descriptive and

can be depicted in many ways. But many researches pointed outthat appropriate recapitulation and summaries of the situation tothe members of the team are the important skill that leaders musthave in emergency conditions (Douglas and John, 1993). The con-cept of ‘Supportiveness of SA building’ is an ability to maintainthe SA level of all members of the team. Statements of the SROto arouse and draw other members’ attention are counted and theratio is used for the measure of supportiveness of SA building, asexpressed in Eq. (3). If the SRO announces the procedures step bystep and explains situations, other members can easily follow whatthe team is doing. So, the total number of main steps, which is theminimally required number of statements for SA building, is usedfor the denominator in the case study.

S = SL

n(3)

where S is the supportiveness of SA building, n the total possi-ble number of supportive statement in the scenario and SL is thesupportive statements that a leader makes.

A part of the dialog shown in Table 3 in emergency conditionsexpresses a clear difference in leaders’ supportiveness to build teamSA. Teams 2 and 3 conducted the same procedure for the LOCA sce-nario. The SRO in Team 2 always gave the number of the step beforeperforming it with clear explanations. On the contrary, the SRO inTeam 3 seldom gave information about what he was performing.Evidently, members in Team 3 often asked the SRO what they hadto do.

3.3.3. Situation awareness skillThoroughness. A basic assumption to build correct SA is to

gather as much good information as possible. Information in EOPsis high quality when emerging conditions match to EOPs. Thus,operators are required to follow highly prescriptive EOPs duringemergency conditions. According to this view, all that is needed forsuccessful performance is that operators read and follow the indi-vidual steps in the EOPs (Roth et al., 1994). Data were counted in amain step weighted manner. This reduces the exaggerated counts ofstep execution. If the importance of all sub-steps is treated equiva-lently, as expressed in Eq. (4), then the effect of a missing importantsub-step in one main step is unwillingly blinded by conductingthe other sub-step in the other main steps. For example, whenthe procedural steps marked with stars in Table 4 are counted inan equivalent sub-steps manner, then the step execution rate is0.5625, based on Eq. (4).∑N

i=1ki∑ni=1ni

= 916

= 0.5626 (4)

where ki is a number of executed sub-steps in the ith step of pro-cedures by operators, ni the total number of sub-steps in the ithstep of the procedure and N is the total number of sub-steps in theprocedure.

In this study, procedural execution rate was counted in the main

step weighted manner, as defined in Eq. (5):∑N

i=1ki/ni

M= (1/3) + 1 + 0 + (1/3) + (5/6) + (1/2)

6= 0.500 (5)

218 H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225

Table 3An example of dialog in LOCA emergency conditions.

Team 2 (LOCA 4) Team 3 (LOCA 5)

Proc. Step Personnel Dialog Proc. Step Personnel Dialog

22 SRO We are on the step 22. Maintain atleast one S/G level between 24.6and 90%.

22 SRO TO! Maintain at least one S/G levelbetween 24.6 and 90%.

TO Yes, Sir. S/G No. 1 level is beingcontrolled at about 46%.

TO Yes, S/G No. 1 60% and No. 2 54 andboth are increasing. I will try tomaintain them.

SRO Yes, confirmed.23 SRO We are on the step 23. Check the

CST inventory exceeds theminimum inventory and refer theCST level curves.

23 SRO Check the CST inventory issatisfied.

TO CST level check. CST No. 1 is 74%,and No. 2 is 76%. They are all abovethe minimum inventory.

TO Yes, the CST inventory check. No. 170%, and No. 2 70%, both are beingmaintained. Above the minimuminventory.

SRO Yes, confirmed.24 SRO We are on the step 24. Check the

temperature of the lowtemperature RCS pipe is below127 ◦C.

24 SRO Check the temperature of the lowtemperature RCS pipe is below127 ◦C.

24.1 RO Temperature of the lowtemperature RCS pipe is check inRCS directory. Temperature has notbeen below 127 ◦C.

RO It has not been reduced.

25 SRO We are on the step 25. Let’s checkthe possibility of bubbles in RCSthrough methods below.

25 SRO Can you check the possibility ofRCS pressure reduction?

SRO Check the possibility of RCSpressure reduction.

RO I will try to reduce the pressure bysub spray valve No. 203.

25.2 RO Yes, I will check the possibility ofpressure reduction by sub sprayvalve No. 203.

RO Yes, pressure reduction is possible.

RO Yes, RCS pressure reduction ispossible.

SRO Yes, confirmed.26 SRO We are on the step No. 26. Check

the off site power is intact.26 SRO Is off site power lost?

EO Yes, from the off site powersystem. . . It is being conducting

EO The off site power is switched fromthe reactor to SFT now. It is not lost.

w

sobrt

3

oumfttvnsupts‘o

Table 4A part of EOPs of the reference plant.

E0

Execution Step Description

� 1 Check the reactor trip1.1 Every control rods’ bottom indicator: on1.2 Reactor trip breaker and bypass breaker: open1.3 Neutron flux: decreasing2 Check the turbine trip

� 2.1 Every low pressure stop valve: close3 Check the connection of emergency AC bus3.1 Emergency AC bus: at least one bus is connected4 Check SI actuation

� 4.1 PZR PRESS LOW SFTY INJ RX TRIP: on4.2 STM LN PRESS STM LN ISO SFTY INJ RX TRIP: on4.3 CTMT PRESS HIGH SFTY INJ RX TRIP: on5 Check main steam line isolation

� 5.1 MFCV: close� 5.2 MFCV bypass valve: close� 5.3 MFIV: close� 5.4 S/G blowdown isolation valve: close� 5.5 S/G sampling valve: close

from off site SFT.

here M is the total number of steps in the procedure.For the former case, there is no change in the value when sub-

tep 5.3 is skipped and sub-step 3.1 is performed instead. On thether hand, the value increases by 0.1388 with the same changey the latter method. This shows that the latter way is better toeflect conduction of the missing sub-step in the main, step 3, inhe example.

.3.4. Decision-making skillTo investigate the decision-making skill, kurtosis and skewness

f the monitoring-control patterns frequency distribution weresed. Kurtosis and skewness were renamed by ‘Latent proceduralistake resistibility’ and ‘latent procedural violation resistibility’

or easier understanding. The decision-making skill is related tohe employing procedures, so these measures together anticipateeams’ capability of dealing with procedures. Each method can beiewed in different ways, e.g. some may argue that kurtosis can-ot exactly represent the ‘procedural mistake resistibility’. It makesense somehow; however, the intention of using statistical meas-res is to see tendencies in team’s behaviors. A usage is fit for theurpose even though the name of the measure does not fully cover

he meaning by notion because definitions of the non-technicalkills are broad. That is why the term ‘latent’ was added to expresspossibility’ to cover remaining semantic uncertainty in the namef the proposed measures.

5.6 Chemical agent injection isolation valve: close6 Check CIS-A

� 6.1 CIS-A signal: generated6.2 CIS A valve: close

ring an

crbtpm

K

wm

cwnbacdovccvovudowktmbsfaksot

srsnioman

S

w

ittmcsv

H.B. Yim et al. / Nuclear Enginee

Latent Procedural Mistake Resistibility. Kurtosis is the statisti-al measure of the ‘peakedness’ of the probability distribution of aeal-valued random variable. One common measure of kurtosis isased on a scaled version of the 4th moment of the data or popula-ion as shown in Eq. (6). A high kurtosis distribution has a sharpereak and longer, fatter tails, while a low kurtosis distribution has aore rounded peak and shorter, thinner tails (Hayter, 2007).

=n∑

i=1

[(Xi − X̄)/s]4

n − 1(6)

here K is the kurtosis, Xi the value of ith sample, X̄ the sampleean, s the standard deviation and n is the total number of samples.To use this statistical measure, frequencies of each monitoring-

ontrol pattern were counted and each monitoring-control patternas assigned values to be distributed. Assigned values were orga-ized in cognitive the activity demanding order, from −2.5 to 2.5,ased on the assumption already mentioned that the cognitivectivity of data-driven monitoring was in between that of pro-edure and knowledge-driven monitoring. A value between twoata-driven monitoring patterns was given to be zero. The purposef assigning them was an intuitive distinction of results, so assignedalues for scaling the x-axis did not affect the results of statisti-al measures use in the study. Also, the range and interval werehosen for the study’s own good, and the data at the ends were con-erged to zero for easy recognition of distribution. When kurtosis ofne team’s monitoring-control pattern data is a positively smalleralue than that of other teams, it means that the former team issing various monitoring-control patterns. Using the knowledge-riven monitoring-control pattern is not harmful to the assessmentf situations. It rather reduces time to complete a given missionhen operators are in correct SA. However, frequent use of the

nowledge-driven monitoring-control pattern possibly shows thathe team considers the procedures as a mnemonic aid. Procedural

istakes can happen under such a circumstance, so the kurtosis cane used for an indicator of possible procedural mistakes. As alreadytated, a higher value of kurtosis does not guarantee better per-ormance. Relative values of this statistic measure are importantnd meaningful to access team’s non-technical skills. Before usingurtosis, mean values of each team’s monitoring-control patternhould be carefully examined when comparing two or more setsf experiments because the kurtosis only shows the peakedness ofhe probability distribution of the random variable.

Latent Procedural Violation Resistibility. Skewness is a mea-ure of the asymmetry of the probability distribution of aeal-valued random variable and 3rd moment of statistics. Values ofkewness can be positive or negative, or zero. According to Eq. (7),egative skew indicates that the tail on the left side of the probabil-

ty density function is longer than the right side and that the bulkf the values (possibly including the median) lie to the right of theean and vice versa. A zero value indicates that the values are rel-

tively evenly distributed on both sides of the mean, typically butot necessarily implying a symmetric distribution (Hayter, 2007).

k =n∑

i=1

[(Xi − X̄)/s]3

n − 1(7)

here Sk is the skewness.When skewness of one team’s monitoring-control pattern data

s a positively bigger value than that of other teams’ data, the formeream is more strongly adherent to the procedures and less likelyo violate procedures. Like kurtosis, mean values of each team’s

onitoring-control pattern should be carefully examined whenomparing two or more sets of experiments because skewness onlyhows the asymmetry of the probability distribution of the randomariable.

d Design 255 (2013) 212– 225 219

4. Case studies

Two sets of experiments were conducted to validate the efficacyof the proposed method. One set consists of LOCA and SGTR casesthat are well understood and known to operators. This set of exper-iments was used to see whether results of the Not-SkiP showedconsistency regardless of changes in variables. The content of theother set is an Interfacing System LOCA (ISLOCA) scenario, which isnot as familiar as the LOCA or SGTR scenario to the operators andthus demands more cognitive work.

4.1. LOCA and SGTR conditions

4.1.1. Data collectionNine sets of LOCA and SGTR emergency operation training situ-

ations of real plant operators in the training center of the referenceplant were recorded. Three operation teams participated in thetraining. Each team performed three scenarios at a one-day inter-val. Teams 1, 2 had the same training schedule; LOCA for the 1stday, LOCA for the 2nd day, and SGTR for the last day of training.A SRO in Team 1 was changed for the last training. Team 3 wastrained SGTR for the 1st and 2nd day and LOCA for the last day oftraining. Thus, there were three independent variables: scenario,training repetition, and members.

4.1.2. Results and discussionFrequency of each monitoring-control pattern and analysis

results of LOCA and SGTR training based on the data frommonitoring-control patterns and verbal protocol analysis, alongwith relative NoT-SkiP values, are summarized in Table 5. The val-ues in blank for supportiveness of SA building are the referencenumbers of the measure; 34 for LOCA and 30 for SGTR training.Sample means and medians are all negative, and values are all sit-uated in procedure-driven monitoring-control patterns. Therefore,kurtosis and skewness can be used as intended.

Changes in scenarios: Experiments 1 and 3 for Teams 2 and 3were used. The graphs of relative value are shown in Fig. 5. In bothteams, relative values of the Not-SkiP showed not much difference;Team 2 showed thorough examination on procedural steps and theSRO was very supportive to form team SA. Compared to membersin Team 2, members in Team 3 stuck to procedures and used theirknowledge less frequently, and thus Team 3 was thought to havemore resistance to procedural mistakes and violations. Communi-cation completion was approximately the same in both Teams 2and 3. Absolute numbers of monitoring-control patterns are trulydependent on what scenario and procedure the team is performingbecause procedures are not made with the same level of temporaldetail. For example, some steps in the SGTR scenario had long timeintervals to execute the next step. This situation might provokeoperators to check their SA to see if they had formed before theyproceeded and could have caused more knowledge-driven mon-itoring actions. This pattern turned up in both teams, so relativeNot-SkiP values were not changed much even with some notablechanges in absolute values. The SRO in Team 2 was very thorough,so he described virtually all the details in the procedures. This ten-dency was shown in relatively large scores for ‘thoroughness’ and‘supportiveness of SA building’. The SRO in Team 2 also used hisknowledge more than the SRO in Team 3 did. This trend impliesthat he may use his knowledge based on procedures. If ‘thorough-ness’ and ‘latent procedural violation resistibility’ of Team 2 werevery much lower than those of Team 3, it is said that Team 2 ispossibly in more a fragile condition to make mistakes than Team 3

is.

If relative values of the NoT-SkiP were not varied much withscenarios, the NoT-SkiP could represent differences in intrinsiccharacteristics of teams. Namely, it is highly probable that Team 2

220 H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225

Table 5Frequency of each monitoring-control patterns and the results of LOCA and SGTR training based on data from monitoring-control patterns and verbal protocol analysis aswell as relative NoT-SkiP values.

Team 1 Team 2 Team 3

LOCA 1 LOCA 2 SGTR 1 LOCA 3 LOCA 4 SGTR 2 SGTR 3 SGTR 4 LOCA 5

Monitor. Pattern 1 31 32 22 34 37 30 28 27 31Monitor. Pattern 2 18 20 16 24 20 18 12 18 18Monitor. Pattern 3 1 1 3 1 1Monitor. Pattern 4 1 3 4 1 1Monitor. Pattern 5 2 1 4 2 5 1Monitor. Pattern 6 1 3 3 2 2 2 3 2Supportive Stat. 17 (34) 9 (34) 13 (30) 27 (34) 30 (34) 30 (30) 12 (30) 13 (34) 8 (30)

Team 1 Team 2 Team 3

LOCA 1 LOCA 2 SGTR 1 LOCA 3 LOCA 4 SGTR 2 SGTR 3 SGTR 4 LOCA 5

Median −2.500 −2.500 −1.500 −2.000 −2.500 −2.500 −2.500 −2.500 −2.500Mean −1.791 −2.009 −1.438 −1.557 −1.817 −1.434 −1.819 −1.613 −1.852S.D. 1.315 0.940 1.603 1.473 1.330 1.590 1.385 1.502 1.276Thoroughness 0.671 0.642 0.567 0.842 0.776 0.853 0.651 0.618 0.585Communication Completion 34.831 30.317 24.957 16.638 13.865 12.479 13.865 16.638 12.479Latent Procedural Mistake 6.531 21.564 2.210 3.110 6.807 1.325 7.164 3.972 9.204Latent Procedural Violation 2.493 3.841 1.704 1.921 2.610 1.509 2.659 2.111 2.908Supportiveness of SA Building 0.395 0.265 0.433 0.794 0.882 1.000 0.400 0.433 0.235

Team 1 Team 2 Team 3

LOCA 1 LOCA 2 SGTR 1 LOCA 3 LOCA 4 SGTR 2 SGTR 3 SGTR 4 LOCA 5

Thoroughness 0.796 0.827 0.664 1.000 1.000 1.000 0.773 0.797 0.685Communication Completion 1.000 1.000 1.000 0.478 0.457 0.500 0.398 0.549 0.500

0.40.71.0

ua

a

Latent Procedural Mistake 0.912 1.000 0.240

Latent Procedural Violation 0.938 1.000 0.586

Supportiveness of SA Building 0.498 0.300 0.433

ses more knowledge than Team 3, and the SRO in Team 3 behaves

rbitrarily in other emergency conditions.

Repetition of training: Experiments 1 and 2 for Teams 1, 2,nd 3 were used. Graphs of relative NoT-SkiP values are shown

Fig. 5. Relative NoT-SkiP values graphs of LOCA an

34 0.316 0.144 1.000 0.184 1.00022 0.679 0.519 1.000 0.550 1.00000 1.000 1.000 0.504 0.491 0.235

in Fig. 6. Latent procedural mistake resistibility for Team 3 showed

quite a reduction. Already mentioned in the changes in the sce-nario section, there are some rather long period steps to go to thenext step in the SGTR procedure. Judging by the verbal protocol

d SGTR training with changes in scenarios.

H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225 221

CA an

abitdiakmlmdivntAw

tbtimant

oosm

Fig. 6. Relative NoT-SkiP values graphs of LO

nalysis, Team 3 was very active at resolving upcoming situationsy using knowledge or their former experience in the second train-

ng. Evidently, the time to complete the training is about 35 min forhe first training and about 28 min for the second training. It can beeduced that Team 3 is very quick to adapt to training. They real-

zed that this scenario could be resolved by following procedures,nd by maximizing the experience by exerting previously acquirednowledge. Team 3 also used more data and knowledge-drivenonitoring in the second training so that the ‘latent procedural vio-

ation resistibility’ dropped. As has already been stated, data-drivenonitoring is not quite related to procedure violation; however, the

istribution of data in the cognitive activity demanding order hasnevitably affected the results of statistical measures. The assignedalues play a role in minimizing this effect. Namely, the higherumber, i.e. knowledge-driven monitoring has more influence onhe measure than the lower number, i.e. data-driven monitoring.part from this, overall shapes for all teams were not changed muchith repetition of training.

Knowledge definitely affects team’s behaviors. Further reduc-ion in latent mistake/violation resistibility and thoroughness cane clearly expected if teams continue to be trained with the sameraining contents. Predicting ‘how quickly teams can learn train-ng contents’ is a possible usage of the NoT-SkiP because using

ore knowledge implies teams learn. However, this usage is notppropriate to the purpose the NoT-SkiP, and thus the NoT-SkiP isot valid when comparing NoT-SkiP results of one team’s repeatedraining.

Changes in members: In Korea, NPPs are run by operating teams

f five members in the MCR: senior reactor operator (SRO), reactorperator (RO), turbine operator (TO), electrical operator (EO), andhift supervisor (SS). It is a common occasion for human resourceanagement in most industries that team members are frequently

d SGTR training with repetition of training.

changed because of vacations, role changes, sick leave, and so on.This kind of work environment is prone to changing and mostly toweakening team skills, including leadership, team work, commu-nication, etc., resulting in poor performance.

Controlled groups are Teams 2 and 3 and the experimental groupis Team 1. The scenario was changed as well as the members. Butthe comparison of the changes in members is viable for the rela-tive values of the NoT-SkiP because the changes in scenario merelyaffect relative values of the NoT-SkiP as presented. As mentionedin the ‘repetition of training’ section, the ‘latent procedural mistakeresistibility’ of Team 3 dropped; however, the value was restoredby changing the scenario. Only the SRO was changed for the lasttraining of Team 1, and other members were intact. The results ofrelative values of the NoT-SkiP are shown in Fig. 7.

Communication completion was developed to measure theproperty of omnidirectional communication and was affected byall participants in the training. But, the other four measures weremainly affected by the dominant player, which was the SRO inthe MCR of NPPs in this experiment. Thus, the overall shape ofrelative values of the NoT-SkiP was significantly changed withthe changes in members. Further experiments with changes inother members are surely necessary. But, with careful thought, achange in a value of communication completion is expected bychanging other members of the team. The changed SRO used hisknowledge and relatively did not follow procedures. Even withthe success of the given SGTR mitigation training, Team 1 withthe changed SRO would cause more mistakes in other complextraining.

As defined, the NoT-SkiP is an evaluation method of non-technical skills of teams, and characteristics of each member haveinfluence on team characteristics. When members are changed inteams, the NoT-SkiP has to be newly conducted.

222 H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225

CA an

4

4

rfsniaedecpnc

4

bLdrdRnfmSnpa1

Fig. 7. Relative NoT-SkiP values graphs of LO

.2. An ISLOCA condition

.2.1. Data collectionThe ISLOCA scenario was developed for understanding what

ole higher-level cognitive activities play in guiding operator per-ormance during complex emergencies. Unlike LOCA or SGTRcenarios training, operators have to deal with situations that areot fully addressed by the procedures in the ISLOCA scenario train-

ng and thus higher-level cognitive activities, such as situationssessment, continue to play vital roles throughout the training,ven with the provision of procedures (Roth et al., 1994). Detailedescription of the scenario is given in NUREG/CR-6028. ISLOCAmergency operation training of real plant operators in the trainingenter of the reference plant was recorded. Nine operation teamsarticipated in the training, and each team performed one sce-ario. Training time is limited to approximately 30 min for a fairomparison.

.2.2. Result and discussionThe purpose of the ISLOCA case study is to find relationship

etween results of the NoT-SkiP and performance. Unfortunately,OCA and SGTR scenarios are so famous in NPPs emergency con-ition training. So, all teams successfully completed given tasksegardless of NoT-SkiP results, and an exact performance relatedescription why teams were doing such ways was hardly made.esults of ISLOCA training and relative NoT-SkiP values for eachon-technical skill are summarized in Table 6. The values in blank

or supportiveness of SA building are the reference number of theeasure. Some are different because some teams formed wrong

A and used different procedures. Sample mean and median are all

egative and are situated in procedure-driven monitoring-controlatterns. Therefore, kurtosis and skewness of all teams can be useds intended. The kurtosis value was negative for Team 1, so Team

is treated as having no resistibility to procedural mistakes.

d SGTR training with changes in members.

As mentioned, NoT-SkiP values were viewed with the perfor-mance of operators using the Operator Performance AssessmentSystem (OPAS) developed by the Halden Reactor Project (Skraning,1998). Only three teams out of nine successfully performed andpointed out the whereabouts of the leakage. Graphs of relativeNoT-SkiP values is shown in Fig. 8. Team 1 showed the worst perfor-mance. It used a procedure that was not relevant to the situation.Educating about the overall structure and importance as well asthe basic usage of the EOPs is recommended for Team 1. Teams2 and 3 had almost identical patterns. But, Team 3 was a littlebit less sufficient in every aspect than Team 2. Except that theSRO in Team 4 was thought to be less supportive to form teamSA than the SRO in Team 5 was, Teams 4 and 5 showed similarpatterns. These four teams tended not to follow procedures in theexact manner in which they should have and all failed the givenmission. They actually had fairly good non-technical skills to copewith emergency conditions. If they use data-driven monitoring orknowledge-driven monitoring based on procedures then they aremore competitive in dealing with any other emergency conditionssuccessfully. Team 2 assessed situations in the wrong direction andused another irrelevant procedure, but they quickly adjusted the SAin the right direction because the situation did not match the pro-cedure they were conducting. In Fact, Team 2 almost succeeded,and possibly has if it had been given a little more time. Team 6was a very good team from the view point of procedural execu-tion and obedience. However, the SRO was so authoritarian thatthere was not much discussion. Worse, he executed procedures allby himself and no one in the team seemed to follow what the SROwas doing. Team 6 used the same irrelevant procedure as Team 2did because of incorrect SA. Unfortunately, this team finally could

not find out that they were in the wrong SA. The teams that suc-ceeded at the given mission within the given time had no commonfeatures but they had evenly reasonable scores. NoT-SkiP valuesfor Teams 7 and 9 imply that they all used their knowledge based

H.B. Yim et al. / Nuclear Engineering and Design 255 (2013) 212– 225 223

Table 6Frequency of each monitoring-control patterns and the results of ISLOCA training based on data from monitoring-control patterns and verbal protocol analysis as well asrelative NoT-SkiP values.

Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 Team 7 Team 8 Team 9

Monitor. Pattern 1 23 43 30 35 34 30 34 41 31Monitor. Pattern 2 4 3 3 3 2 2 5 3 3Monitor. Pattern 3 2 1 4 4 2Monitor. Pattern 4 3 3 1 1Monitor. Pattern 5 1 1 3 3Monitor. Pattern 6 6 2 1 1 3 1 1Supportive Stat. 7 (35) 12 (47) 6 (36) 13 (39) 17 (36) 5 (44) 15 (36) 10 (36) 18 (36)

Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 Team 7 Team 8 Team 9

Median −2.500 −2.500 −2.500 −2.500 −2.500 −2.500 −2.500 −2.500 −2.500Mean −1.167 −1.926 −1.738 −1.900 −1.905 −2.294 −2.000 −2.088 −2.184S.D. 2.082 1.474 1.605 1.514 1.624 1.067 1.261 1.203 0.213Thoroughness 0.366 0.579 0.445 0.554 0.607 0.549 0.725 0.464 0.637Communication Completion 40.112 44.107 31.919 18.301 23.252 11.092 21.613 31.057 47.675Latent Procedural Mistake −0.405 5.260 2.957 4.929 4.997 28.418 10.397 12.420 20.431Latent Procedural Violation 1.116 2.472 1.970 2.413 2.494 5.131 3.062 3.373 4.170Supportiveness of SA Building 0.200 0.255 0.167 0.333 0.472 0.106 0.417 0.278 0.500

Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 Team 7 Team 8 Team 9

Thoroughness 0.505 0.799 0.614 0.765 0.838 0.757 1.000 0.641 0.879Communication Completion 0.841 0.925 0.670 0.384 0.488 0.233 0.453 0.651 1.000Latent Procedural Mistake 0.000 0.185 0.104 0.173 0.176 1.000 0.366 0.437 0.719

0.0.

47

ogatc

Latent Procedural Violation 0.217 0.482 0.384

Supportiveness of SA Building 0.400 0.511 0.333

OPAS Score 28 43 38

n the facts. If their knowledge was not sufficient, they tended to

ather more information before they made decisions for controlctions. Team 9 showed the best NoT-SkiP value among the nineeams. Especially, the SRO was very informative and willing to dis-uss. According to OPAS scores that are shown in Table 6, Team 1

Fig. 8. Relative NoT-SkiP values

470 0.486 1.000 0.597 0.657 0.813667 0.944 0.213 0.833 0.556 1.000

48 42 48 50 57

showed the worst performance and had the lowest OPAS score as

well as the worst preparedness. Obviously, Team 9 showed the bestperformance and had the highest OPAS score and NoT-SkiP values,and Team 8 was not thought to have good non-technical skills foremergency conditions. The overall shape of NoT-SkiP values tells

graphs of ISLOCA training.

2 ring an

tttbeBfnccfttSe

5

fmsmOstapbsifTpuapwtcdcosariwftsbpnhete

R

A

A

24 H.B. Yim et al. / Nuclear Enginee

hat Team 8 will more likely fail or have trouble while conductinghe other emergency training. Actually, Team 8 succeeded to findhe leakage point and conducted the exact remedy for it. Judgingy the high OPAS score, Team 8 consists of operators with highxpertise, or they have previous experience of similar conditions.ut, this team has to be trained for all four non-technical skills

rom the view point of the NoT-SkiP. The rest of the teams wereot distinguishable by OPAS scores. Teams 4 and 5 could have suc-eeded, or Team 7 could have failed based on OPAS scores. As wean see in the results, the OPAS alone cannot exactly represent per-ormance of operators because these operators work in teams andhe non-technical skills are important aspects in the depiction ofhe dynamics of team behaviors. In this sense, results of the NoT-kiP can be used as a compensatory measure for performance undermergency conditions.

. Conclusions

NPPs are run by operation teams. A distinct feature of teamsrom individual operators is ‘working together’. In other words,

embers of teams must be equipped with the good non-technicalkills to handle given tasks. Thus, training has to consider improve-ent in both the technical and the non-technical skills of teams.peration teams in NPPs have been mainly trained the technical

kills so far. Thus, this study proposed the concept of ‘Non-echnical Skills Preparedness’ for emergency condition handling,nd suggested the method to evaluate operation teams’ emergencyreparedness in the non-technical skills aspect called ‘NoT-SkiP’ased on a literature survey. Six monitoring-control patterns wereuggested and applied to a video analysis to use visual training datan a stochastic manner. The verbal protocol analysis was also per-ormed to find out differences in the non-technical skills of teams.hree teams participated for the LOCA and SGTR case study and theyerformed three emergency training scenarios each. NoT-SkiP val-es were not very much affected by the changes in the scenariosnd by the repetition of training. However, it is found to be inap-ropriate to use the NoT-SkiP when one team is repeatedly trainedith the same training scenario. The changes in the members of

he team, especially the dominant player, contributed much to thehanges in values of the NoT-SkiP. The ISLOCA scenario, whichemands a higher level of cognitive activity, was used for the secondase study. Nine teams participated and training was performednce for each team. Teams with relatively good NoT-SkiP resultshowed successful completion of the given mission, and teams withny deficiency in the non-technical skills, judging by the NoT-SkiPesults, failed to identify the leakage point. This kind of result givesnsights and plausible explanations why performance of teams

ith approximately the same level of the technical skills varies,or example, knowledge-driven monitoring appeared to be helpfulo access situations and even reduced time to complete given mis-ions, but was sometimes very harmful when monitoring was notased on the correct situations or procedures, providing that therocedures could correctly describe the situations. The NoT-SkiP isot a perfectly proven solution to describe the needs for ‘what andow to train the non-technical skills’. It surely needs more backupxperiments. Nevertheless, the NoT-SkiP gives trainers clues howo assess various training results and plan for the next training moreffectively.

eferences

huja, M.K., Carley, K.M., 1999. Network structure in virtual organizations. Organ.Sci. 10, 741–757.

rmstrong, M., 1997. A Handbook of Personnel Management Practice, reproducedin Personnel in Practice. Blackwell Business, Oxford, UK.

d Design 255 (2013) 212– 225

Arora, S., Miskovic, D., Hull, L., Moorthy, K., Aggarwal, R., Johannasson, H., Gautama,S., Kneebone, R., Sevdalis, N., 2011. Self vs expert assessment of techni-cal and non-technical skills in high fidelity simulation. Am. J. Surg. 202,500–506.

Belton, S.,2001. CRM training in the nuclear industry. In: 3rd CRM Users GroupWorkshop. University of Aberdeen, UK.

CAA, 2002. Flight Crew Training: Cockpit Resource Management (CRM) and Line Ori-ented Flight Training (LOFT), CAP 720. Civil Aviation Authority: Safety RegulationGroup, Hounslow, Middlesex, U.K.

Cacciabue, P.C., 2004. Application of HERMES in Training: A Human Factors Coursein the Aviation Domain, Guide to Applying Human Factors Methods. Springer-Verlag London Limited, pp. 190–229.

Cannon-Bowers, J.A., Salas, E., Grossman, J., 1991. Improving tactical decision-making under stress: research directions an applied implications. In:International Applied Military Psychology Symposium, Stockholm, Sweden.

Cepin, M., 2008a. Importance of human contribution within the human reliabilityanalysis (IJS-HRA). J. Loss Prev. Proc. Ind. 21, 276–368.

Cepin, M., 2008b. DEPEND-HRA—a method for consideration of dependency inhuman reliability analysis. Reliab. Eng. Syst. Saf. 93, 1452–1460.

Chang, Y.H.J., Mosleh, A., 2007. Cognitive modeling and dynamic probabilistic simu-lation of operating crew response to complex system accidents. Part I. Overviewof IDAC Model. Reliab. Eng. Syst. Saf. 92, 1997-1013.

Chung, Y.H., Yoon, W.C., Min, D.H., 2004. An extended framework for analyzingteam communication in advanced nuclear power plant. In: Proceedings of theIFAC/IFIP/IFORS/IEA symposium, Atlanta, GA.

Cioffi-Revilla, C., O’Brein, S., 2007. Computational analysis in US foreign and defensepolicy. In: Nau, D., Wilkenfeld, J. (Eds.), The First International Conference onComputational Cultural Dynamics. College Park, MD, pp. 26–36.

Crichton, M.T., Flin, R., 2004. Identifying and training non-technical skills of nuclearemergency response teams. Ann. Nucl. Energy 31, 1317–1330.

Damme, D.V., Kjær-Hansen, J., Amat, A.-L., 1998. Human factors in the investigationof accidents and incidents. In: European Air Traffic Control Harmonization andIntegration Program, HUM.ET1. ST13.3000-REP-02.

Dien, Y., 1998. Safety and application of procedures, or ‘how do ‘they’have to use operating procedures in nuclear power plants?’. Saf. Sci. 29,179–187.

Douglas, K.H., John, E.K., 1993. Systematic Evaluation of Nuclear Operator Team SkillsTraining: A Progress Report.

Endsley, M.R., 1995. Measurement of situation awareness in dynamic systems. Hum.Factors 37 (1), 65–84.

Endsley, M.R., 2000. Theoretical understandings of situation awareness: a crit-ical review. In: Endsley, M.R., Garland, D.J. (Eds.), Situation AwarenessAnalysis and Measurement. Lawrence Erlbaum Associates, Mahawah, NJ,pp. 22–23.

Hayter, A.J., 2007. Probability and Statistics for Engineers and Scientists, 3rd ed.Thomson Brooks/Cole, US.

Helmerich, R.L., Merritt, A.C., Wilhelm, J.A., 1999. The evolution of crew resourcemanagement training in commercial aviation. Int. J. Avian Psychol. 9 (1), 19–32.

Henttonen, K., 2010. Exploring social networks on the ream level—a review of theempirical literature. J. Eng. Technol. Manage. 27, 74–109.

Hollnagel, E., 2005. Human reliability assessment in context. Nucl. Eng. Technol. 37(2), 159–166.

IAEA, 2002. Safety Reports Series No. 75 INSAG-15, Key Practical Issues in Strength-ening Safety Culture, Vienna.

INPO, 1993. Control Room Teamwork development Training: Course Administrationand Facilitation Guide. National Academy for Nuclear Training, Atlanta, GA.

Kim, S.K., Byun, S.N., Lee, D.H., Jeong, C.H., 2009. Development of a crew resourcemanagement training program for reduction of human errors in APR-1400nuclear power plant. J. Eng. Soc. Kor. 28 (1), 37–51.

Kim, S.H., Park, J.K., Han, S.Y., Kim, H.J., 2010. Development of extended speechact coding scheme to observe communication quality of human operators ofnuclear power plants under abnormal conditions. J. Loss Prev. Proc. Ind. 23, 539–548.

Kim, S.H., Park, J.K., Kim, Y.J., 2011. Some insights about the characteristics of com-munications observed from the off-normal conditions of nuclear power plants.Hum. Facors: Eng. Manuf. Serv. Ind. 21 (4), 361–378.

Kolaczkowski, A., Forester, J., Lois, E., Cooper, S., 2005. Good Practices for Imple-menting Human Reliability Analysis (HRA). Sandia National Laboratories,NUREG-1792, US NRC.

NEA#05334, 2004. Nuclear Regulatory Challenges Related to Human Performance.NEA.

NEI 09-07, 2009. Fostering a Strong Nuclear Safety Culture. Nuclear Energy Institute,Washington.

O’Connor, P., O’Dea, A., Flin, R., Belton, S., 2008. Identifying the team skills requiredby nuclear power plant operations personnel. Int. J. Ind. Eng. 38, 1028–1037.

Operational Performance Information System for Nuclear Power Plants(opis.kins.re.kr).

Rasmussen, J., Goodstein, L.P., 1985. Decision Support in Supervisory Control. Risø-M-2525, Risø National Laboratory, Denmark.

Reason, J., 1990. Human Error. Cambridge University Press, New York.Reinartz, S.J., Reinartz, G., 1992. Verbal communication in collective control

of simulated nuclear power plant incidents. Reliab. Eng. Syst. Saf. 36,245–251.

Roth, E.M., Mumaw, R.J., Lewis, P.M., 1994. An Empirical Investigation of OperatorPerformance in Cognitively Demanding Simulated Emergencies, NUREG/CR-6028, US NRC.

ring an

S

S

S

S

H.B. Yim et al. / Nuclear Enginee

alas, E., Kendall, D.L., 2004. Measuring team performance: a review of current meth-ods and consideration of future research. Adv. Hum. Perform. Cogn. Eng. Res. 5,307–326.

alas, E., Prince, C., 1999. A method for enhancing crew resource management train-

ing. Hum. Factors 41 (1), 161–172.

cott, J., 2000. Social Network Analysis: A Handbook, 2nd ed. Sage Publication Ltd.,London.

harma, B., Mishra, A., Aggarwal, R., Grantcharov, T.P., 2011. Non-technical skillsassessment in surgery. Surg. Ontol. 20, 169–177.

d Design 255 (2013) 212– 225 225

Sheridan, T.B., 1981. Man–Machine Systems; Information, Control and DecisionModels of Human Performance. MIT Press, Cambridge.

Skraning Jr., G., 1998. The Operator Performance Assessment System (OPAS), HWR-538.

Wang, M.L., Chen, W.Y., Lin, Y.Y., Hsu, B.F., 2010. Structural characteristics, process,and effectiveness of cross-functional teams in hospitals: testing the I-P-O model.J. Hi-tech. Manage. Res. 21, 14–22.

Yang, H.L., Tang, J.H., 2004. Team structure and team performance in IS development:a social network perspective. Inform. Manag. 41, 335–349.