Cognitive Work Analysis Modeling for Tactical Decision Support

Bruce A. Chalmers Robert D.G. Webb Roy Keeble Maritime Information & Combat

Systems Defence R&D Canada - Atlantic Dartmouth, Nova Scotia, Canada

(902) 426-3100x390 Bruce.Chalmers@drdc-rddc.gc.ca

Humansystems Incorporated 111 Farquhar Street Guelph, ON, Canada

(519) 836-5911 webb@humansys.com

Humansystems Incorporated 111 Farquhar Street Guelph, ON, Canada

(519) 836-5911 roy.keeble@ns.sympatico.ca

Abstract The 21st century is witnessing significant and growing challenges in naval warfare. Developments in the capabilities of maritime threats, larger numbers of smaller, newer threats that can be rapidly deployed, and an increasing requirement for agile and adaptive responses to rapid change at all levels of Command and Control are among the many factors widely anticipated to increase demands on command decision making. It is anticipated that operators could be supported through various technological interventions incorporating capabilities such as data fusion and decision support systems. Although these are promising for enhancing performance, incorporating such technologies is extremely challenging. This paper is concerned with ongoing R&D that is exploring a particular cognitive engineering approach to the problem of designing advanced computer-based tools to assist the Operations Room team of a HALIFAX Class frigate with tactical Command and Control. We illustrate some aspects of the approach in tactical picture compilation work. It is based on developing work models to permit formulating a model of support for data and information fusion aids. Aspects of the work being modeled include the dynamic information processing in individual operators’ picture compilation tasks, the information flow and integration among the operator team, and their goals for acquiring and maintaining situation awareness. 1. Introduction A variety of changes in naval operations are producing increased demands on shipboard Command and Control (C2). These include improvements in sensor, weapon, and information technologies that lead to the collection and processing of increasing quantities of data about the tactical situation and a requirement to achieve higher combat power and efficiencies under increasingly time-compressed conditions. There are also enhancements in the capabilities of conventional threats and the emergence of other newer types of threats to consider. An example of the latter would be the increasing number of smaller, rapidly deployable threats such as shoulder launched missile systems and suicide bombers. The nature of naval operations themselves is also continually changing. This is evidenced, for example, by the shift from blue water to littoral operations. Littoral warfare poses difficulties such as:

• geographical constraints that reduce the size of the battlespace and increase platform vulnerabilities,

• increased traffic congestion that adds uncertainties in identification and deconfliction, • increased ability of the threat to strike at any time with little advance warning (e.g., attacks

from coastal defence systems), and • degraded performance of sensor and guidance systems due to land clutter. With the growing number of operations other than war (e.g., humanitarian aid, Maritime Interdiction Operations (MIOs), and Leader Interdiction Operations (LIOs) as part of anti-terrorist measures), naval operators also have to deal increasingly with unanticipated threat tactics and highly ambiguous tactical situations. Operators must also work in an increasingly coordinated manner within coalition task groups made up of multi-national participants with a variety of policies, skill sets and equipment capabilities. Finally, the co-evolution of organization, doctrine and technology underlying the Revolution in Military Affairs will impose increasing demands for agile and adaptive responses to rapid change at all C2 levels. For example, the shift from platform-centric to network-centric warfare [Cebrowski and Garstka, 1998] calls for the capability to self-synchronize forces in real time, from the bottom up, at the same time respecting top down organizing principles of warfare such as unity of effort, the commander’s intent, and rules of engagement. These changes result in increasingly higher operational tempos and call for faster C2 cycles to execute the mission. They also increase scenario complexity and impose greater demands on operators for performing their tasks in a timely manner in a large and growing variety of situations. The range of potential situation types spans, at one end of the spectrum, those that are familiar and routinely encountered, to unfamiliar, but anticipated ones in the middle of the spectrum, to those at the other end of the spectrum, involving both unfamiliar and unanticipated events. The latter type of situation calls, in particular, for increasing adaptation and novel problem solving in real time in time-critical conditions. A sudden crisis situation, or one involving an asymmetric terrorist threat, where there is limited knowledge of the threat’s tactics and capabilities, would be examples of this latter type of situation. In practice, with the evolution in the types of missions, we could expect operations to increasingly involve elements that from one moment to the next touch the entire breadth of this spectrum. More than ever, support tools are now needed to help naval tactical teams deal effectively with their cognitive and collaborative work demands over this broad and evolving range of situations. It is anticipated that operators could be supported through various advanced technological interventions incorporating capabilities such as data fusion, decision support systems, information visualizations, collaborative or integrative work aids and displays, intelligent interfaces, and knowledge management. For example, the data fusion literature provides a variety of algorithmic techniques to combine data about entities in an environment to estimate or predict their states [Steinberg and Bowman, 2001]. Information visualizations provide interactive, visual representations of abstract data to amplify human cognitive processing and

reduce demands on operators in performing their perceptual and cognitive tasks [Card et al., 1999]. In one response to the challenge of determining how to provide effective tools with these types of advanced capabilities, there has been an increased emphasis in Defence R&D Canada in the areas of data fusion and decision support to the Operations Room team of a Canadian Navy HALIFAX Class frigate, with the goal of improving the team’s battlespace awareness, and increasing decision speed and accuracy. For example, one project is developing a technology demonstrator to investigate algorithms and displays for multi-source data fusion based on track data from the existing sensor suite for the frigate [McArthur et al., 1999]. There are in fact a number of possible areas of C2 work on the HALIFAX Class frigate which appear ripe for design interventions that help enhance operational performance. Considering, for example, the work that operators perform as they conduct a watch, they could be helped, both individually and as a team, to access and manage the increasing volumes and types of data needed to perceive, assess, and comprehend the tactical situation and resolve situation ambiguities. Another example, also related to the watch period, would be to augment the operators’ capacity to shape the unfolding tactical situation in accordance with their mission objectives, and increase their ability to act decisively and effectively. However, independent of the choice of work areas to focus on, a critical design challenge, given the pivotal role played by the human in C2 and the pace and range of the continually evolving demands of the work, must be the question of how to effectively support operators over the full spectrum of situation types they can be called upon to deal with. In particular, how should operators be supported to create flexible, knowledge-based problem solving teams, while harnessing their individual and combined competencies and adaptive capabilities to the full? This paper is concerned with some aspects of ongoing R&D that is exploring a particular cognitive engineering approach to the problem of designing advanced computer-based tools to assist the Operations Room team of a HALIFAX Class frigate with tactical C2. The high-level goal of the research is to identify key elements of a principled approach for deriving the needed design insights [Chalmers, 1998], [Chalmers et al., 2000], [Chalmers et al., 2001], and, furthermore, to use the approach to capture detailed design requirements, in an iterative manner, in the form of prototypes of support tools. The approach we are exploring incorporates two frameworks. The first is a system engineering framework for developing and testing design hypotheses for advanced support capabilities. It spans, in an opportunistic, nonlinear manner, various activities that are aimed at developing an increasing understanding of the environment’s cognitive and collaborative work demands (which we refer to simply as its work demands), and in using this to determine operator support requirements to improve performance in light of these demands. Using the framework effectively involves developing incremental prototypes as a means of decomposing complex aspects of the work’s demands into manageable portions, identifying increasingly robust solutions for these various portions of the workspace, and integrating lessons learned from early prototypes into a final prototype. An essential, distinctive component of the framework that underpins this exploratory process is a work analysis to explicitly model work demands as a means of deriving the support requirements. This emphasis on work analysis is the reason we refer to the approach as work-centred. The work analysis itself is based on a second framework, the Cognitive Work Analysis (CWA) framework [Vicente, 1999]. CWA is a layered, constraint-based framework for conducting a work analysis that is

particularly suited to modeling and analyzing work constraints in complex, dynamic work environments. We have already presented results at previous CCRTS Symposia from two studies exploring this general approach. The first study [Chalmers et al., 2001] looked at the first of CWA’s layers. More specifically, this involved conducting a Work Domain Analysis of the HALIFAX Class frigate. This led to developing a multi-level abstraction hierarchy to structurally represent the broadest limits of the work domain’s constraints that create tactical action opportunities on the frigate. The second study [Chalmers et al., 2000] provided, in the form of a case study, a demonstration of the utility of the CWA framework for generating novel computer-based decision support to satisfy operator demands on the HALIFAX Class frigate. Support requirements were captured in a dynamic storyboard incorporating various information visualizations. Both studies found that CWA is certainly applicable to the C2 work environment of the HALIFAX Class frigate. The second study, in particular, provided a significant first opportunity to demonstrate the design potential of the approach, in an end-to-end manner that spanned knowledge acquisition, modeling and analysis, requirements generation, and design activities in the system engineering framework. In this paper, we first review the general approach for generating design concepts and decision support tools for this complex work environment. We then illustrate some of its aspects specifically in the context of tactical picture compilation work within the frigate’s Operations Room, where we are developing a variety of CWA-based models to represent the intrinsic cognitive demands of the work. In particular, we describe here the general modeling approach and the method we have adopted to perform the knowledge acquisition needed to build the work models. The eventual aim of this phase of the research is to use these models in other ongoing work to formulate a model of effective operator support for tactical picture compilation as a basis for exploring specific design interventions in this workspace. By its nature, the approach avoids forcing a technological option onto the work environment that results in unexpected negative consequences for operational performance. 2. Need for a joint operator-machine perspective in design Shipboard C2 presents unique challenges for the design of tools to support the demands of the cognitive and collaborative work involved. The Operations Room team needs to integrate and understand the significance of a large amount of context-dependent information, including own ship’s capabilities and the capabilities and the intentions of friendly and hostile parties, and act in a coordinated manner under a variety of constraints. One standard way of reducing work demands on operators in a military context, which has significant implications for personnel training and mission preparation activities, is to rely heavily on doctrine, standard operating procedures, and operations orders to communicate plans and procedures. Various studies of decision-making strategies in complex naval C2 settings have found that experienced naval operators also make extensive use during operations of a recognition-based approach to make decisions under conditions of time pressure, high risk, and ambiguity. This particular type of decision-making strategy is captured in a number of

Naturalistic Decision Making (NDM) models. In such models, operators essentially use their experience to recognise the situation and act based on matching courses of action to the situation to obtain a satisfactory and viable course of action rather than employing more cognitively demanding strategies such as analytically generating and comparing alternatives to derive an optimal solution [Kaempf et al., 1993], [Kaempf et al., 1996]. Although this does not yet appear to have received attention in the military decision support literature, it is also conceivable that other psychologically plausible models of bounded rationality that humans use to cope with the demands of limited time, knowledge, and computational power, may also naturally occur in this environment. For example, [Gigerenzer et al., 1999] has recently made a case for a variety of fast and frugal heuristics that are more formal than existing NDM models, which humans employ to limit their search for information, using easily computable stopping rules and easily computable decision rules, to quickly make adaptive choices in real environments in some ecologically rational manner. According to Gigerenzer, a fast and frugal heuristic is ecologically rational in an environment to the extent that it is adapted to the structure of that particular environment. However, even with such ways of reducing complexity by training and planning for it, and the human resorting to strategies to cope with the work demands in real time, there is much in a tactical situation that cannot be predetermined and must be established and dealt with as events unfold. In fact, as already indicated, the shipboard team can expect to increasingly face unanticipated or anomalous situations that do not conform to previous experience. Such situations may require other types of knowledge and reasoning strategies, to be able to resolve situation ambiguities, understand the situation, and decide and act effectively. In turn, this may require new, emergent, and adaptive patterns of problem-solving and decision-making behaviours, involving a variety of team members working together to socially construct their knowledge and understanding of the situation in real time in response to anomalous events. We have previously observed that tactical C2 has many characteristics in common with those of other complex sociotechnical work systems [Chalmers, 1998]. These characteristics are significant because they impact the nature of the work demands on operators in these domains. Examples of such work systems that are receiving wide attention in the research literature include process control plants, aviation cockpits, surgical environments, and engineering design. The distinguishing characteristics of this type of system include the presence of uncertainty, dynamism, team work, high stress, high risk, interaction with open environments that are subject to unanticipated disturbances, large amounts of data to process, high potential for sensory overload, imperfect data, human-environment interactions mediated via computers, and complex multi-component and context-sensitive decision making. Most approaches to developing automated tools in such cognitively demanding environments effectively start from a premise of replacing human involvement as a means of increasing data and information processing capacity and speed, reducing operator workload, or eliminating the opportunity for human error. There is an implicit assumption that the new automation can be substituted for human action without any larger impact on the human-computer interaction in which that action occurs. However, investigations in a number of applied domains have shown

that this assumption, commonly referred to as the substitution myth [Sarter et al. 1997], is not valid. More typically, introduction of advanced automated tools has shifted the human role to one of monitor, exception handler, and manager of automated resources. This can lead to many new types of problems. For example, performance decrements can result from automation-induced complacency (over-trust) and biases that increase, rather than decrease, errors as operators come to rely on automated cues as a heuristic replacement for their own vigilant information seeking and cognitive processing [Parasuraman et al., 1993]. Other examples include breakdowns in the interaction between human operators and their automated tools, particularly as situations in the dynamic environment with which operators interact move from routine cases to unusual or exceptional ones, or even ones that have not been anticipated by designers. In the latter type of situation, in particular, tools may even be called upon, often without the operator’s knowledge, to perform outside their competency envelopes. The necessary interaction to deal with this spectrum of situations is not supported by most systems. Most systems are designed to be precise and powerful autonomous agents that are not equipped with communication capabilities, with comprehensive access to the outside world, or even with complete knowledge about the tasks, and the context of those tasks, in which they are engaged. Rather than realising the putative benefits of such tools, this leads instead to an escalation of cognitive and coordinative demands on human operators [Woods and Patterson, 2001] as they use their expertise to adapt and tailor responses to deal with the unusual circumstances. In fact, [Woods and Patterson, 2001] observe that such situations already naturally impose greater demands on operators for knowledge, monitoring, attentional control, information, and communication among team members. Therefore, the demands of the situation are only further aggravated if the workload associated with using the automated system, and understanding its assumptions and outputs, is at the same time creating new, additional workload burdens. These observations underscore the need in designing tools to support cognitive work to adopt a joint operator- or team-machine system perspective as the critical unit of analysis. With this perspective, design requires a sound understanding of the cognitive demands on operators and how those demands vary dynamically with the context of the tactical situation. This is the motivation for the systems-based, cognitive engineering approach we have adopted in our work. 3. Framework for exploratory design In previous work [Chalmers, 1998], we have drawn attention to the ill-structured nature of the problem of designing decision support systems for this work environment. [Klein, 1996] has suggested the need for nonlinear problem-solving frameworks when dealing with ill-structured problems, to allow complete descriptions of problem detection, problem representation and problem solving activities, and their inter-relations. Compared with approaches for solving well-structured problems, this reflects a greater focus on explicitly dealing with the ill-structured nature of the problem in an exploratory manner, by continually modifying goal definition and

problem representation throughout the problem solving process, rather than just at the beginning, based on continual feedback and insights gained from intermediate, tentative “solutions”. At a high level, exploratory design can be conceptualized as an opportunistic, dynamic evolution of multiple interacting activities, with continual goal and activity refinement, based on developing an increasing understanding of two key spaces: the problem space and the solution space. This is shown below in Fig. 1. The problem space is the work environment, made up of the work domain, its workers or operators, their organization, policies, doctrine, tactics and procedures. The work domain is the system being controlled, independent of any particular operator, automated controller, event, task, goal, or interface [Vicente, 1999]. The solution space is the artifact space or design space.

ConceptDemonstrator

CWA modelingCWA modeling

Evaluation

Design

Implementation

RequirementsGeneration

KnowledgeAcquisition

KnowledgeSources

Requirements

PhysicalArtifact

Conceptual Artifact

Models

DomainKnowledge

ProblemSpace =

Work Environment

Solution Space = ArtifactSpace

Supp

ort H

ypot

hesi

s

Eva

luat

ion

Res

ult

Feedback

Figure 1. Exploratory framework showing one possible activity trajectory A review of existing (essentially linear) systems engineering frameworks suggests that candidate activities in such a framework are: domain knowledge acquisition, analysis, requirements generation, design, implementation, and evaluation. Figure 1 also illustrates one activity trajectory, showing the activity nodes and their potential linkages in terms of their inputs and outputs. In general, trajectories exploring the problem and solution spaces could be considerably more chaotic. A key output is a demonstration of some advanced support concept. This is not an operational prototype of the concept, however, but a means of capturing and instantiating design knowledge in a tangible, validated artifact as a basis for further exploration and development of an operational prototype.

It is important to describe the dynamic, nonlinear aspects of the framework, since these are its critical features. This refers to its opportunistic, bootstrapping, incremental and evolutionary nature. Several opportunistic sequences are conceivable, motivated by a number of factors. These include where in the “life cycle” a particular concept is under investigation, the intent of a concept evaluation, and the need to repeatedly shift attention between the problem space and the solution space. This happens as design hypotheses, generated through analysis in the problem space, lead to detection and framing of the problem, fleshing out design goals for tentative solutions, testing hypotheses in the solution space, feedback, and so on. It is also important to formalise the details of the analysis component, labeled as CWA modeling in Fig. 1, that make the framework work-centred and the design approach model-based. We do this in section 5 for the specific application considered in this paper of picture compilation work on a HALIFAX Class ship. Continuously shifting attention between the two spaces is akin to Woods’ idea that designs to support cognitive work are really only hypotheses about how artifacts can shape cognition and collaboration [Woods, 1998]. Concept demonstrators are the result of applying a scientifically accepted hypothesize-and-test paradigm. The bootstrapping characteristic of the framework relates to the fact that each step builds incrementally on the insights and products of earlier ones. Furthermore, although the various activities are shown in generically labeled activity nodes in Fig. 1, their specific details (e.g., techniques, strategies) on a given pass are linked to the actual approaches that need to be adopted on that pass and are not predetermined. This explains the evolutionary nature of the framework. The framework is a basis for developing advanced aiding concepts without losing sight of the domain’s characteristics and the requirements of its operators. Knowledge acquisition plays a key role by producing domain knowledge for other activities. This can involve a variety of techniques, including interviews or knowledge elicitation sessions with human subject matter experts (SMEs), verbal protocols, naturalistic observations of training or actual work situations, and consultation of procedural and system design documents (for the existing system), depending on the sources of the desired knowledge. Working with SMEs to construct work scenarios, or to elicit feedback using storyboards, mock-ups or other types of artifacts, or conducting a sea trial, can involve aspects of both evaluation of an existing concept and an opportunity to pursue further knowledge acquisition, leading to further analysis and problem representation. Finally, we note the separation of knowledge acquisition from analysis in the framework. This is to distinguish between domain description, which is the aim of knowledge acquisition, and modeling, which is the purpose of analysis. 4. Tactical picture compilation We are currently using the system engineering framework to explore design issues for tools to support tactical picture compilation work within the HALIFAX Class frigate’s Operations Room. In naval C2, the process of compiling and understanding the integrated tactical picture is known as tactical picture compilation. This is a highly cognitive process, involving a variety of operators, within and between C2 levels, and dealing with a variety of types of data and

information of varying degrees of timeliness and quality. It produces a continually updated product, referred to as the Recognised Maritime Picture (RMP)1.

CO ORO SWC ASWC

HMS

SCS

SPSC

SIG1 SIG2

STIR A

STIR B RT2TSRT1SAC

On-Watch Tech

ESM

EWS

CIO

TAS1

TAS2

CHART TABLE

STATEBOARD

ST

AT

EB

OA

RD

ST

AT

EB

OA

RD

Helo

JMCIS/GCCS

LINK

OOWIntelOPTASK

FileCCR

Counterpart Other ship

OTC

Manuals

TACPAC

SAC STATEBOARD

LSO

Current File

FLEX

AAWC

ASWC

ASuWC

EWC

HEC

ORS

FCS

TASS

C2ISCheatSheet

Figure 2. Operations room layout and information sources

It is anticipated that operator support tools will incorporate advanced processing capabilities, based on data and information fusion. These tools would help them build and maintain an integrated picture derived by fusing Above Water Warfare, Underwater Warfare, Tactical Data Link, and Wide Area Picture data inputs. They are expected to form part of advanced computer-based aids and integrative work aids and displays, to augment the capabilities of the frigate’s Command and Control Information System (C2IS). The goal is to enhance operator performance in responding to the demands posed by current and future warfare scenarios. On a HALIFAX Class frigate, the C2IS forms the core of the combat system that provides the war-fighting functionality of the ship. It gives human operators access to services for building and displaying a tactical picture and using the ship’s fighting capabilities for threat neutralisation. Most of the cognitive demands of the high-level information processing within C2 are left to the Command Team, supported by the combat personnel in the various warfare areas. This can impose significant cognitive demands on operators to integrate large amounts of available information and compile the tactical picture. In this paper, we illustrate several critical aspects of

1 The term RMP and related or overlapping terms such as the COP (Common Operating Picture), MTP (Maritime Tactical Picture), and RAP (Recognised Air Picture) are not always used consistently either in the literature or by members of the operational community. In this paper, the RMP is taken to include mission relevant air, surface and sub-surface components whatever the source platform (i.e. land, air or sea-based).

the work involved in building the RMP by referring to the current operator organization. This helps ground the discussion and illustrate some aspects of picture compilation work that we are modeling. The inner portion of Fig. 2 provides an overview of operator positions in the Operations Room. The Operations Room Officer (ORO), the ship’s tactical coordinator located in what is referred to as the “back row”, manages and disseminates the data from all available sensors and information sources to build the RMP. The ORO is aided by two warfare directors, the Sensor Weapon Controller (SWC) and the Assistant Sensor Weapon Controller (ASWC). The SWC’s primary focus is on the Above-Water Warfare (AWW), comprising both the Anti-Air Warfare (AAW) and Anti-Surface Warfare (ASuW) areas, and the ASWC’s is on the Anti-Submarine Warfare (ASW). In turn, these directors are supported by teams that handle various types of sensor data that contribute to building the picture. Located around this inner portion of Fig. 2 are various external positions and systems that could be involved in providing information to or receiving information from internal positions. For example, the AAWC, ASWC, ASuWC are Task Group positions, shown on the outside of Fig. 2, that could be located on other platforms and would be responsible for coordinating the associated warfare areas of the task group as a whole. Both passive and active sensor systems can be involved in providing data inputs in the air, surface, and sub-surface warfare domains. These systems are controlled by electronic sensor operators (on the left side of the Operations Room in Fig. 2), combat information operators (in what is known as the “front row”), and acoustic sensor operators (on the right side). As well as gathering and processing information available through their displays and other sources (e.g., data links, intelligence sources, the OPTASK, stateboards), operators communicate over an internal communications network to actively exchange information, coordinate decisions and actions, and even passively assimilate information from listening on the nets. A general objective in building the RMP is to detect, localize, track, identify, and classify all air, surface, and sub-surface traffic within the ship or task group’s area of operational interest (AOI). For this, sensor information is filtered and combined with other sources of information and knowledge as it flows through the various operator levels from the sensor-level operators to the members of the Command Team in the back row, who manage the execution of the mission as a whole. Although this description refers to specific processes that operators engage in, individually or as a team, to detect and amplify the information on a contact-by-contact basis, the RMP is better thought of as the situation picture needed to support any and all aspects of tactical operations over an AOI of a maritime commander. This extends the usual approach to thinking about the RMP as consisting only of information about the location and identity of entities in an AOI to potentially include other tactically relevant aspects of the situation. In particular, the RMP is used to develop and maintain awareness of the battlespace and make tactical decisions to achieve the mission. In practice, therefore, the requirements for the RMP are highly dependent on the mission and the goals of the commander in charge of that mission. This means that the information the RMP may need to represent is not absolute. This is analogous to the context-sensitivity problem of information defined by [Woods, 1991]. What constitutes information to a commander at any given moment, as well as the basis for interpreting this information, is a

dynamic relation between the data, the world the data refers to, and the commander’s expectations, intentions, and interests. The ORO will have a number of context-dependent picture compilation goals that depend on the mission (see section 6), and will monitor the progress of his/her various teams in building the picture in accordance with these goals. Although many goals can be concurrently active, which ones have priority at a given moment depends on the specific situation at that moment. This can be interpreted as the ORO having at any moment a number of goals in the foreground that receive specific attention, with others that remain in the background but which are nonetheless continually attended to at a lower level of priority. A variety of cues can shift attention among the goals. These include: new intelligence information, a verbal report from another ship, a visual sighting from the OOW (Officer of the Watch) on the ship’s bridge that raises a question about a track, some piece of knowledge about possible enemy tactics, the expected return time of the ship’s helicopter that has been operating over the radar horizon, and so on. The ORO will use these priority shifts among goals to focus his/her team while not ignoring the other lower priority goals. S/he may also determine that various goals are not being met and take corrective measures (e.g., by querying one of his/her directors). The ORO can also become involved in sorting out uncertainties on the level of an individual contact, particularly in situations of high workload that require adaptive worksharing. As the situation changes, priorities change and the ORO supervises more closely the related parts of the picture. Here, the ORO is looking at the whole picture to understand what the changes mean. Not only are individual tracks important, but also the various formations, relative positions, geographic position, and other things. The ORO wants the picture to tell everything, so s/he examines what the sensors and team are presenting, and pushes them to tell more. 5. Cognitive Work Analysis modeling framework We are exploiting a specific Cognitive Engineering framework known as Cognitive Work Analysis (CWA) [Vicente, 1999] in the work modeling component in Fig. 1. CWA is a systems-based framework for work analysis in complex, sociotechnical work environments. Our reasons for considering CWA are related to the characteristics of the work environment being explored and the limitations of other work analysis approaches for modeling work demands in the presence of these characteristics that limit their modeling power [Chalmers et al, 2000]. For example, CWA offers a conceptual capability to model work demands of operators that are a consequence of a great deal of behavioural variabilities and emergent behaviours in complex sociotechnical systems [Vicente, 1995]. Examples of such behaviours include: • the need to adapt responses to unanticipated events arising from unforeseen disturbances that

impact the state of the work (e.g., see [Vicente and Rasmussen, 1992]);

• operators’ discretionary need to resolve many degrees of freedom in particular situations, based on their own subjective performance criteria (e.g., see [Rasmussen, 1986, Rasmussen, 1997]); and

• inadequacy of routine problem-solving methods for dealing with novel situations and the consequent need to reinterpret and reevaluate existing knowledge, usually involving almost ad hoc interactions among a variety of team members (e.g., see [O’Neill, 1996]).

These behaviours can manifest themselves in several work aspects, including the way it is shared among operators in high-load situations, the strategies employed in processing information depending on factors like expertise, perceived work load, perceived cognitive burden of using one strategy over another, and the specific information that is needed for these strategies [Vicente, 1999]. O’Neill’s reference in [O’Neill, 1996] to the evolving nature of artifacts that are produced and exchanged by a group of workers to represent their shared understanding of ill-structured problems that they confront is another specific example.

Activities

Work domain

Require information about

Act on

Figure 3. Activity-work domain relationships

CWA modeling is organized according to a variety of behaviour-shaping work constraints. It is consistent with a design philosophy that tools to support cognitive work should not overconstrain its performers, but rather help reduce cognitive workload by making more “visible” (e.g., selection, extraction, processing) the task-relevant semantics of the data and information they need to perform the work over the entire range of situations they have to deal with. Five layers of work constraints are usually distinguished, beginning with a Work Domain Analysis (WDA), followed by analyses of Control Tasks, Strategies, Social-Organization and Cooperation, and Operator Competencies [Vicente, 1999]. We concentrate here only on the first two analyses. Also, following [Sanderson, 1998], we broaden the scope of the second analysis to a more general Activity Analysis (AA). This is motivated by the modeling requirements for tactical picture compilation. A WDA integrates a variety of types of knowledge about the work domain, including its intentional knowledge (its purposes), causal knowledge (interactions between subsystems), and structural knowledge (its physical parts and characteristics). It produces an event-independent description of the domain, usually as an abstraction hierarchy [Vicente, 1999].

Operators engage in activities on the work domain to achieve the picture compilation goals. The relation of activities to the work domain is illustrated above in Fig. 3. Mapping operator activities onto the various elements of the work domain provides a means of extracting their data and information requirements [Vicente, 1999], as well as representing the context of those requirements. However, describing intrinsic constraints on activities in decision-making terms requires other forms of representation. This is the role of an AA.

Table 1. Goal structure. (COI = Contact of Operational Interest)

Generic Goal Structure AOI Integrate ship picture with overall Area of Operational Interest Entire Ship Integrate across air / surface / sub- surface domains for ship

Air Picture

Surface Picture

Sub-surface Picture

Warfare Domain Ensure complete & up to date Ensure complete & up to date Ensure complete & up to date COI Relations Integrate all COI.

Relate to mission. Integrate all COI. Relate to mission.

Integrate all COI. Relate to mission.

Each Contact Detect, localize, track, identify, classify

Detect, localize, track, identify, classify

Detect, localize, track, identify, classify

Given the importance of understanding how operator goals are formed and attended to in tactical picture compilation work, and the sensitivity of these goals to context, we have used two types of representation in the AA. The first is the decision ladder (DL) [Vicente, 1999], which focuses on what needs to be done and not who needs to do it. It allows representing the variabilities and information processing short-cuts due to skill-based (SBB), rule-based (RBB), or knowledge-based behaviours (KBB) in the contextually derived cognitive sequences used by operators to perform control tasks. This model template represents both information processing activities and their resulting states of knowledge [Vicente, 1999]. The second representation provides a framework for capturing picture compilation goals. We needed a representation that can be linked to DL representations and permit various characteristics of those goals to be captured, e.g., goal concurrency, foreground goals versus background goals, the various levels of situation awareness (SA) that goals relate to. We introduced one that (as far as we know) is new to the CWA literature. It is a goal structure that currently distinguishes two dimensions among picture compilation goals. The first dimension maps goals onto various levels of a goal hierarchy according to their degree of specificity, or temporal or spatial coverage. On some of these levels, warfare areas are also further distinguished. The second dimension addresses the three levels of SA in Endsley’s sense [Endsley, 1995], consisting of perception, comprehension, and projection, that the goals relate more specifically to. Some of the details of this structure are shown in Table 1; however, for simplicity, it does not distinguish picture compilation goals according to their SA levels. The various types of models for the two Work Domain and Activity analyses are illustrated below in Fig. 4. The reader can refer to [Chalmers et al., 2001] for details on the work domain models shown below in Fig. 4 only in high-level form.

ACTIVATION:detection of need for action

ALERT

EXECUTE:coordinate manipulations

OBSERVE:information and data

FORMULATE PROCEDURE:plan sequence of actions

PROC-EDURE

INFO TASK

IDENTIFY:present state of system

DEFINETASK:select appr. change of system cond.

INTERPRET:consequences for current task, safety, etc.

EVALUATE: performance criteria

SYS.

STATE

GOAL

STATE

AMBI-

GUITY

ULT.

GOAL

Perceived in

terms of task

Perceived assystem state

System state directly

associated with task

Preset response

SBB

RBB

KBBKBB

RBB

ACTIVATION:detection of need for action

ALERT

EXECUTE:coordinate manipulations

OBSERVE:information and data

FORMULATE PROCEDURE:plan sequence of actions

PROC-EDUREPROC-EDURE

INFO TASK

IDENTIFY:present state of system

DEFINETASK:select appr. change of system cond.

INTERPRET:consequences for current task, safety, etc.

EVALUATE: performance criteria

SYS.

STATE

GOAL

STATE

AMBI-

GUITY

ULT.

GOAL

Perceived in

terms of task

Perceived assystem state

System state directly

associated with task

Preset response

SBB

RBB

KBBKBB

RBB

Model of picture

compilation goals

A goal corresponding to a specific contact

HALIFAX External Entity

Purpose

Principles

Processes

Physical Capability

Physical Form

Purpose

Principles

Processes

Physical Capability

Physical Form

Natural Environment

Principles

Processes

Physical Capability

Physical Form

Decision Ladder

Abstraction Hierarchy

ACTIVITY MODELS WORK DOMAIN MODELS

Model ofPicture Compilation Goals

Figure 4. Work domain and activity models in tactical picture compilation

6. An example To test the modeling framework in a preliminary validation exercise, we had an SME, an ORO with operational experience in the Adriatic Sea in support of the United Nations Maritime Interdiction Operations (MIO) against the Former Republic of Yugoslavia, recount one scenario from this experience and map his picture compilation goals onto the generic framework in Table 1. Tables 2, 3, and 4 present the result of the mapping. One incident in the validation exercise involved a contact that was observed initially displaying normal behaviour, when suddenly its behaviour began to deviate from the anticipated. Actions then had to be taken to resolve the uncertainty about the contact. The various models shown in Fig. 4 were used to map the ORO’s reasoning as his attention shifted from a general picture compilation activity to a more intensive goal-directed focus on the specific contact. At the level of the contact, his reasoning was mapped onto a decision ladder associated with the contact (shown emanating from the goal structure in Fig. 4). In addition, his various information requirements were successfully mapped onto the work domain models.

Table 2. Example of the ORO’s perception goals

Situation Awareness: Perception Goals AOI Detect & track every aircraft, surface vessel & submarine

Entire Ship Present all tracks in common picture. Show all relevant mission, territorial, geographical context.

Air Picture

Surface Picture

Sub-surface Picture

Warfare Domain

Detect & track, organically or with LINK, every contact in the AOI. Show all relevant context (geographical, territorial, mission)

COI Relations Track and display all air and surface contacts. Display waterspace management areas and non-friendly sub-surface contacts in relation to these.

Each Contact Set up relevant radar, sonar, communication equipment and team(s) to detect all contacts within AOI. Know position, course speed, IFF signature.

Table 3. Example of the ORO’s comprehension goals

Situation Awareness: Comprehension Goals AOI Identify and classify every contact

Entire Ship Understand relationships among contacts (i.e. to enemy or friendly forces) and between contacts and geographical points

Air Picture

Surface Picture

Sub-surface Picture

Warfare Domain

Identify all contacts. Display in geographical context. Understand relevance, including threat level.

COI Relations Identify cooperation & relationships among suspect contacts, in relationship to geographical / territorial and mission constraints

Each Contact Use information to identify & classify all contacts. Understand what contact is doing at the moment.

Table 4. Example of the ORO’s projection goals

Situation Awareness: Projection Goals Area of Interest Understand what is expected to happen or may be developing within the overall

AOI Entire Ship Anticipate incoming units or those traversing AOI. Recognise patterns among

suspect forces that may indicate a challenge to friendly forces. Project possible movements / intent of COI.

Air Picture

Surface Picture

Sub-surface Picture

Warfare Domain

Recognise potential air threats building. Understand own forces coverage potential in terms of time distance relationships among units (air, surface, sub-surface) and boundaries (territorial limits, geographical points, etc.).

COI Relations Understand potential threat / hazard (e.g. collision) to surface forces from friendly, neutral, suspect contacts. Understand role of friendly units.

Each Contact Understand characteristics of contacts and threat potential (speed, armament, etc), and time / space implications.

7. Work to date The model building approach outlined in this paper draws on several building blocks. The project team itself represents several perspectives: system design, human factors and cognitive psychology, and naval operations from an ex-ORO in a HALIFAX Class frigate. This core team has worked together to systematically draw insights and data from a large number of other serving naval personnel with relevant experience of Operations Room activities. The pooling of knowledge among the project team has proved to be one of the most fruitful aspects of the work and emphasizes the iterative learning process underlying such modeling work. The work also builds on earlier scenario-based studies of HALIFAX Class Operations Room activities conducted by different members of the team, including: an analysis of Operations Room deficiencies; a Cognitive Task Analyses of the ORO position [Matthews et al., 1999]; and a WDA of a frigate [Chalmers et al., 2001]. Ongoing work has examined different data capture and model building approaches. The data capture goals for this model building work are outlined in Table 5 below. Our approach has been to use experienced operators representing a small cross section of the Operations Room team working through scenarios of varying complexity. Two forms of data capture have been investigated. The first approach presented different warfare sub-teams, led by an ORO, with a detailed paper-based description of a mission scenario involving passage of the frigate through busy coastal waters while responding to various types of threats. The team(s) then walked through a series of static screen configurations representing the state of contacts around them from the point of view of their position, and described their own and the team’s tactical picture compilation activities in the terms outlined in Table 5 below. While successful in some respects, and undoubtedly economical, this approach was rejected as too static, too abstract and resulting in hypothetical answers rather than interpretation of realistic real-time responses based on dynamic contextual cues. Subsequent work has investigated the potential of using different team training simulators to present more realistic, dynamic, contextually rich scenarios, and to gather both interview data during pauses in the scenario and ongoing real-time response data for later analysis. The training simulator chosen, NCOT (Naval Command Operator Trainer), provides realistic simulation and recording of all screen display formats and communications among linked workstations, as well as facilities to simulate a wide range of environmental, sensor, and contact features and behaviours. Scenarios can be paused and re-started during play and review. To date, a two-hour scenario of routine and threat response events has been prepared for a four-position team consisting of the ORO and the AWW team (dealing with air and surface tactical picture compilation and threat responses). A pilot trial has established data capture and subsequent data reduction approaches to conduct the modeling outlined in this paper. The scenario comprises standard background context and events, over which different mission events may be laid. During the scenario, all screen-based and audio communications are captured for later analysis, as well as video data of team activity as a whole.

Table 5. Picture compilation data needs

Item Picture Compilation Related Data Tactical picture compilation goals

Operator goals for building their tactical picture and (where multiple simultaneous goals are being pursued) relationships among the goals and sub-goals. Identify individual and team picture compilation goals that remain constant throughout the scenario. Decompose goals into greater detail such as sector, contact type, picture quality, etc.

Information needs Specific information needs associated with picture compilation goals. Information sources For identified information needs, document actual or potential sources for

that information. Information transfers For picture compilation goals, transfers of information (by any means,

including verbal communication, message, C2IS, etc.) between two or more positions.

Processing activities For picture compilation goals, Operator information processing activities in terms of their mental representations and information transformations, computations, and decisions.

Individual and team strategies

For picture compilation goals, individual and team strategies for selecting data and information inputs, transforming data, and making decisions.

Instances of collaboration

Instances of two or more Operators collaborating in any way (goals, when, who, roles, what information, was information processed separately or jointly)

During the trial, observers at each position record operators’ activities (including interactions with other operators) over selected segments covering either routine events or unusual or ambiguous events intended to provoke knowledge-based behaviours. After each segment of interest the scenario is frozen. During the freeze, observers interview operators at their workstations to construct individual decision ladders for their immediately preceding activities, and to establish the data outlined in Table 5. At the end of the entire scenario, the operators, as a team at their workstations, review each segment, and work with the observer team to build a team decision ladder identifying inter-relationships among picture compilation goals among the team, and information transitions between team members. 8. Design implications The global goal of this work is to support design of upgrades to the C2IS for HALIFAX Class Operations Room teams that result in tactical mission-related decisions of all kinds being made more quickly and more accurately, under all circumstances. A related goal may be to optimize manning levels. To achieve these goals, designers need to understand the picture compilation goals of operators and how operators achieve those goals, as individuals and as a team. Understanding of the latter is required in two senses. One sense is to achieve a specific understanding of how operators achieve their goals using the existing system in order to permit identification of shortcomings and upgrades that eliminate them. A second, more fundamental, sense is to gain a generic understanding of human factors involved in goal achievement so that this knowledge can be used

to generate entirely new approaches to system design, using more effective principles for cooperative work involving human teams and other components of future systems. The modeling efforts described here are expected to support the design effort in several ways. • Identification of decision- and information-seeking trajectories during representative picture

compilation activities at the individual and team levels. Consideration of decision ladders will identify short cuts to speed decision making or make it more reliable or timely, or redirect overlapping efforts among team members to better dovetail team efforts.

• Identification of information aggregation strategies to fulfill picture compilation goals: what information is sought, from where, and how it has to be combined to serve the goals of different operators involved in different steps in the picture compilation process. This will permit consideration of ways of making the aggregation process easier for operators. This could be in terms of making data and information more easily accessible (e.g., information visualizations) or by pre-compiling data currently presented in a fragmented, widely distributed form around the Operations Room. Other instances might include the following: ways that aggregated data could be made more comprehensible for operators, especially to facilitate recognition-primed decision making in relation to planned mission issues; fostering more realistic mental models of real-world issues in relation to mission achievement, especially under high workload conditions; and providing scanning prompts where stress-related tunnel vision may reduce the probability of effective attention strategies, or for known shortcomings in human cognition, such as a bias against consideration of disconfirming cues.

• Identification of the information goals and needs of different classes of operators: those in different positions (with different responsibilities), and those with different capabilities (e.g., novice versus experienced operators). Customization options may be considered – such as context-based prompts (as required), and customized aggregation of selected data sets for different missions.

• Identification of multi-tasking and supervisory requirements; for example, how individual operators can be better supported to manage multiple contacts at different stages of the detect-to-resolve cycle or how supervisors can better scan the overall picture to identify and correct shortfalls in picture compilation efforts among members of their team.

The design process for complex data presentation and decision support systems for this type of work environment can be described as an iterative learning process within which specialists from diverse domains seek to combine their specialty knowledge to achieve their design goals. In many such teams, understanding of the users’ role in the domain of application is, relatively speaking, superficial. The data acquisition and modeling efforts described in this paper represent an approach to provide a design team with a means of understanding the cognitive aspects of how people work together in a frigate’s operations room to compile and use the tactical picture to make mission related decisions. Importantly, it will also provide a flexible modeling approach for use during the design cycle. This will facilitate examination of the likely impact of any proposed interventions as well as providing a readily accessible archive of behavioural examples to revisit whenever, inevitably, the need to learn more arises during the design process.

9. Conclusions This paper described a work-centred, model-based approach for designing effective tools to support Command and Control on a HALIFAX Class frigate. It also described the models that are being used to model tactical picture compilation work on the frigate. The approach shows considerable promise for providing the detailed insights designers need to develop technical solutions to support demanding cognitive and collaborative work. The next phase of this research will use a navy trainer to conduct the data collection and extend the existing models of picture compilation work, as a basis for design. 10. References [Card et al., 1999] S.K. Card, J.D. Mackinlay, and B. Shneiderman. Information Visualization: Using Vision to Think. San Francisco: Morgan Kaufmann Publishers, Inc., 1999. [Cebrowski and Garstka, 1998] A.K. Cebrowski and J.J. Garstka. Network-Centric Warfare: Its Origin and Future. Naval Institute Proceedings, 1998. [Chalmers et al., 2000] B.A. Chalmers, J.R. Easter, and S.S. Potter. Decision-centred visualisations for tactical decision support on a modern frigate. Proceedings of the Command and Control Research and Technology Symposium, Monterey, CA, 2000. [Chalmers et al., 2001] B.A. Chalmers, C.M. Burns, and D.J. Bryant. Work domain modeling to support shipboard Command and Control. Proceedings of the 6th International Command and Control Research and Technology Symposium, Annapolis, MD, 2001. [Chalmers, 1998] B.A. Chalmers. On the design of a decision support system for data fusion and resource management in a modern frigate. Proceedings of Sensor Data Fusions and Integration of the HUMAN, pp. 2-1 – 2-13, NATO System Concepts and Integration Panel Symposium, 1998. [Endsley, 1995] M.R. Endsley. Toward a theory of situation awareness in dynamic systems. Human Factors, Vol 37, No. 1, pp. 32-64, 1995. [Gigerenzer et al., 1999]. G. Gigerenzer, P.M. Todd, and The ABC Research Group. Simple Heuristics that Make us Smart. Oxford:Oxford University Press, 1999. [Kaempf et al., 1993]. G.L. Kaempf, S. Wolf, G., and T.E. Miller. Decision making in the AEGIS combat information center. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, pp. 1107-1111, 1993. [Kaempf et al., 1996]. G.L. Kaempf, G. Klein, G., M.L. Thordsen, and S. Wolf. Decision making in complex naval command-and-control environments. Human Factors, Vol. 38, No. 2, pp. 220-231, 1996.

[Klein, 1996] G. Klein. Nonlinear aspects of problem solving. Information and Systems Engineering, Vol. 2, pp. 195-204, 1996. [Matthews et al., 2001] M.L. Matthews, R.D.G. Webb, and D.J. Bryant. Cognitive Task Analysis of the HALIFAX Class Operations Room Officer. Report by Humansystems Inc. to Defence and Civil Institute of Environmental Medicine, Downsview, Canada, 1999. [McArthur et al., 1999] B. McArthur, A. Ashley, É. Bossé, and D. Beevis. The Command Decision Aid Technology (COMDAT) Project. Unpublished manuscript, 1999. [O’Neill, 1996] J. O’Neill. The role of problem-solving in C3I systems. Proceedings of the 2nd International Symposium on Command and Control Research and Technology, Market Bosworth, Warwickshire, UK, pp. 348-355, 1996 [Parasuraman et al., 1993] R. Parasuraman, R. Molloy, and I.L. Singh. Performance consequences of automation-induced “complacency”. The International Journal of Aviation Psychology, Vol, 3, No. 1, pp. 1-23, 1993. [Rasmussen, 1986] J. Rasmussen. Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. New York:North-Holland, 1986. [Rasmussen, 1997] J. Rasmussen. Merging paradigms: decision making, management, and cognitive control. In R. Flin, E. Salas, M. Strub, and L. Martin (Eds.). Decision Making Under Stress: Emerging Themes and Applications. Hants:Ashgate, 1997. [Sanderson, 1998] P. Sanderson. Cognitive Work Analysis and the analysis, design and evaluation of human-computer interactive systems. Proceedings of OzCHI’98, pp. 220-227, Adelaide, Australia, 1998. [Sarter et al., 1997] N.B. Sarter, D.D. Woods, and C.E. Billings. Automation surprises. In G. Salvendy, (Ed.), Handbook of Human Factors and Ergonomics, 2nd edition, pp. 1926-1943, New York: Wiley, 1997. [Steinberg and Bowman, 2001] A.N. Steinberg and C.L. Bowman. Revisions to the JDL data fusion model. In D.L. Hall and J. Llinas (Eds.), Handbook of Multisensor Data Fusion. Boca Raton:CRC Press, 2001. [Vicente and Rasmussen, 1992] K.J. Vicente and J. Rasmussen. Ecological Interface Design. IEEE Transactions on Systems, Man, and Cybernetics, Vol. 22, No. 4, pp. 589-606, 1992. [Vicente, 1995] K.J. Vicente. Task Analysis, Cognitive Task Analysis, Cognitive Work Analysis: What’s the difference? Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting, pp. 534-537, 1995.

[Vicente, 1999] K.J. Vicente. Cognitive Work Analysis: Towards Safe, Productive, and Healthy Computer-Based Work. New Jersey: Lawrence Erlbaum Associates, 1999. [Woods and Patterson, 2001] D.D. Woods and E.S. Patterson. How unexpected events produce an escalation of cognitive and coordinative demands. In P.A. Hancock and P.A. Desmond, (Eds.), Stress, Workload and Fatigue, pp. 290-304, New Jersey: Lawrence Erlbaum Associates, 2001. [Woods, 1991] D.D. Woods. The cognitive engineering of problem representations. In G.R.S. Weir and J.L. Alty, (Eds.), Human-Computer Interaction and Complex Systems, pp. 223-243, San Diego: Academic Press, 1991. [Woods, 1998] D.D. Woods. Designs are hypotheses about how artifacts shape cognition and collaboration. Ergonomics, Vol. 41, pp. 168-173, 1998.