Fidelity Considerations for Simulation-Based Usability …folk.ntnu.no/oleanda/papers/Fidelity considerations... · 2009-10-03 · Fidelity in Simulation-Based Usability Assessments

Fidelity Considerations for Simulation-Based Usability

Assessments of Mobile ICT for Hospitals

Authors:

Yngve Dahl (corresponding author)

Telenor Group Business Development & Research

Telenor ASA

7004 Trondheim, Norway

Phone: +47 905 27 892

E-mail: [email protected]

Ole Andreas Alsos

Department of Computer and Information Science

Norwegian University of Science and Technology


Phone: +47 915 44 825


Dag Svanæs

Department of Computer and Information Science

Norwegian University of Science and Technology


Phone: +47 918 97 536


Fidelity in Simulation-Based Usability Assessments

2

Abstract: We have conducted controlled usability evaluations of mobile ICT for hospitals.

As part of these evaluations, clinicians have acted out mobile work scenarios and used the

systems to solve related tasks. The evaluations show that relevant usability factors go beyond

that of graphical user interfaces. Some of these usability factors only show up when the real-

world context of use is replicated in the laboratory to a high degree of fidelity. The

complexity of the context of use for mobile ICT in hospitals has motivated us to explore

training simulation fidelity theory. Based on a review of the training simulation literature, we

identify a set of fidelity dimensions through which training simulations often are adjusted to

meet specific goals. We argue that the same mechanisms can be used in usability assessments

of mobile ICT for hospitals. We substantiate our argument by using the identified set of

fidelity dimension in a retrospective analysis of two assessments. The analysis explains how

the fidelity composition contributed to the identification of relevant usability factors.

Keywords: Clinical information systems, mobility, simulation, simulation fidelity, usability

assessment, user-centered design.


3

1. Introduction

This paper investigates the concept of fidelity and its role in controlled usability assessments

of mobile information and communication technology (ICT) for hospitals. Within HCI

literature, fidelity has traditionally been used to describe the extent to which a software

application reproduces visual appearance, interaction style and functionalities during

evaluation (Virzi, 1989; Virzi, Sokolov, & Karis, 1996). In conventional PC-based usability

testing, the physical and social aspects of the use situation are of little concern. Interaction

with PC-based systems is highly uniform and static with regard to the physical and social

aspects of the use situation—one user sitting in front of a PC, with his or her attention

directed mainly on the computer screen.

As interaction with ICT becomes more mobile in nature and take place in dynamic work

settings, such as hospitals, the old standards for usability testing no longer hold. Clinical

work is characterized by extensive mobility, rapid context shifts, changing work priorities,

and close interaction between different actors (Bardram & Bossen, 2005; Reddy, Dourish, &

Pratt, 2006; Sørby, Melby, & Nytrø, 2006). As a use setting for ICT, these characteristics

make hospitals very different from office environments – the prototypical use environments

for desktop computers. In contrast to relatively static office environments, the usability of

mobile ICT supporting clinical work is also likely to depend on factors beyond the GUI and

software solution per se, including physical and social aspects of the use situation. Such

external factors cannot be addressed through conventional usability testing in laboratories

designed for evaluating desktop computer applications.

Over the last five years we have done controlled usability tests of a number of mobile ICT

systems for hospitals, both formative evaluations of prototype systems and summative

evaluations of products ready for deployment. In these usability assessments hospital workers


4

have used the products to solve specific tasks during enacted scenarios. The scenarios have

been acted out in a laboratory that has been custom made to mimic a hospital ward section.

The laboratory has movable walls in a 10 x 8 meter room that allows for full-scale

simulations of different hospital settings. The rooms are equipped with patient beds, chairs

and tables to create a high level of realism. We have consulted health workers in this process.

Gradually, we have become aware that our approach has strong elements of simulation – a

method associated with skill training in various high-risk industries, such as aviation, naval

shipping, and health care. Fidelity is a central concept in simulation research and in the

design of training programs. In contrast to the case for HCI, simulation research often

describes fidelity as a multi-dimensional concept, encompassing various aspects of the

research setting. In training simulations, the various fidelity dimensions are often tailored to

fulfill specific goals (Liu, Macchiarella, & Vincenzi, 2008).

We argue that to assess usability of mobile ICT for complex use settings, such as hospitals,

evaluators need to carefully consider the fidelity of the research setting vis-à-vis the actual

performance context. For simulations to work as an effective tool in the design process, it is

critical to identify the right level of simulation fidelity. Similar to the case for training

simulations, the fidelity of simulation-based usability assessments can be adjusted to achieve

targeted trials that help participants focus on specific aspects of the simulation experience.

Drawing on two earlier formative usability evaluations, we first aim first to show that fidelity

in usability evaluations of mobile ICT is a concept that extends beyond the prototype GUI

being evaluated. Particularly, when addressing hospital settings, where the technology is

likely to be used as part of work activities requiring manual labor with hands and feet in

addition to high situational awareness, elements of the use context become vital components

of the total system being simulated.


5

Our second objective of this paper is to demonstrate how fidelity theory from training

simulation research can be applied as a guiding framework for composing targeted usability

evaluations of mobile ICT for complex use settings, such as hospitals. Research on training

simulations has identified various fidelity components or dimensions that can be adjusted to

achieve goal-specific training. Based on a set of relevant simulation fidelity components we

will conduct a retrospective analysis of the main findings from the two usability evaluations.

The analysis provides a rational for why the two assessments succeeded in evoking relevant

user responses and behavior. This is conceptualized in a simulation acceptance model.

The main purpose of the paper is to draw attention to the relevance of simulation fidelity

theories in the design of formative evaluation addressing the usability of mobile ICT for

clinical work.

The general structure of the paper is as follows. In Sect. 2, we will provide a definition of the

term simulation-based usability assessment, and describe the motivation behind the approach.

Sect. 3 will briefly outline the limited understanding of the fidelity concept in HCI vis-à-vis

simulation research. In Sect. 4, we will draw some parallels and distinctions between training

simulations and usability assessments. Next, in Sect. 5, we will present relevant theory from

simulation research, and identify a set of fidelity dimensions through which simulation-based

training often is adjusted. Sect. 6 will present the design and results from two former

simulation-based usability assessment, in which mobile ICT for hospitals have been

evaluated. In Sect. 7, we analyze how the fidelity configurations for the assessments helped

us identify relevant usability factors, and propose a simulation acceptance model. Sect. 8 will

shed light on issues of relevance to the design of simulation-based usability assessment, and

identify some limitations of the current study. Finally, a brief summary along with

concluding remarks is provided in Sect. 9.


6

2. Simulation-Based Usability Assessments

2.1. Definition

A simulation is broadly defined by Beaubien and Baker (2004) as "a device that attempts to

re-create characteristics of the real world". Simulations can generally be divided into two

categories. The first category consists of computer-generated models of a system or an

environment. This models can be digitally simulated according to predefined rules of

operation (McGrath, 1995). The second category includes simulations in which a particular

system is imitated in the physical world with representative actors of that system as

participants. In the current work we will focus on simulations fitting the latter category.

The term simulation-based usability assessment is used here to refer to a usability test in

which the design concept being evaluated is employed by end users enacting in constructed

work scenarios in natural-like physical environments. Generally, such evaluations are

designed to reproduce specific contextual factors of the real-world setting in a controlled

manner. Kjelskov and Skov (2007) refer to such approaches as in sitro evaluations.

A scenario-based usability assessment fulfills the general features of simulations identified by

Gagné (1962):

• It attempts to represent the real situation in which actions are performed

• It provides the simulation participant with some control over the situation.

• It deliberately omits some aspects of the real situation.

A fundamental challenge related to all simulations, then, is what aspects of the performance

context can be omitted without compromising the validity of the results.


7

2.2. Motivation

Below, we will provide a brief account of the methodological and practical issues that has

motivated us to follow a simulation-based approach.

2.2.1. Methodological Concerns

The question of which research setting that is optimal for studying mobile usability has been

addressed in earlier studies (Kjeldskov, Skov, Als, & Høegh, 2004; Kaikkonen, Keklinen,

Cankar, Kallio, & Kankainen, 2005; Nielsen, Overgaard, Pedersen, Stage, & Stenild, 2006;

Rogers, Connelly, Tedesco, Hazlewood, Kurtz, Hall, Hursey, & Toscos, 2007). The studies,

however, offer different views.

One explanation for the ambiguity in the results is that the question of optimal research

setting essentially is a question of which research criteria you want to maximize. McGrath

(1995) identifies three desirable, but conflicting, criteria when studying phenomena or events

in social and behavioral sciences. These are generalizability of research evidence, precision

of measurements, realism of the situation or the context being studied. McGrath argues that

different research methodologies prioritize different criteria. Simulation-based usability

assessment, which is the focus in the current investigation, corresponds to what McGrath

classifies as experimental simulation. Experimental simulation is a compromising strategy in

which one attempts to partly maintain the control associated with laboratory experiments,

while retaining some of the realism associated with experiments conducted in the field. Some

of the main methodological benefits of evaluating mobile usability in simulated contexts

rather than in the field include:

• Laboratory settings provide immediate access to relevant situations. Relevant use

situations can be created “on demand” and repeated.


8

• Laboratory settings allow evaluators to “pause” situations and collect feedback from

participants while the episode is still fresh in their minds.

• Laboratory settings offer the possibility to do detailed video and audio recordings.

2.2.2. Practical and Ethical Concerns

In addition to the methodological motivation described above, there are also practical and

ethical reasons why simulation-based approaches can be valuable when studying mobile

usability. Field evaluations may have and obtrusive effect on the care situation being studied,

and discretion need to be shown with regard to sensitive patient information. The latter may

prevent the use of audio and recording equipment, which are valuable data collection tools

when studying usability.

Simulation-based usability assessments, then, can be seen as a pragmatic approach when field

evaluations are infeasible due to security and safety reasons.

3. The Fidelity Concept in HCI

Within HCI, fidelity has traditionally been thought of in engineering terms, referring to the

extent to which a prototype system is perceived as authentic or realistic by end users (Virzi,

1989). Thus, in the case of high-fidelity prototypes, a user should experience little or no

differences between the prototype and the end product. In this sense, the fidelity concept has

been considered a one-dimensional feature limited to describing aspects of the prototype.

This understanding of the fidelity concept corresponds with traditional models of human-

computer interaction. Typically, these models denote the user and the computer as two

information-processing units in closed-loop system (Kaptelinin, 1996). Phenomena existing

outside the loop are excluded. Consequently, it has made little sense to talk about the fidelity

of the physical and social use context when evaluating usability of software running on


9

desktop computers. Recreating the prototypical use context of desktop computers for

usability evaluation purposes is generally a trivial issue.

As further discussed in Sect. 5, the conceptualization of fidelity as a one-dimensional feature

is contradicted in simulation research literature. Within simulation research the concept is

used not only to describe the accuracy of devices used during trials but also denotes the

accuracy of the context in which the trial are held. This understanding of fidelity is highly

relevant in the design of simulation-based usability assessments.

4. A Comparison of Training Simulations and Usability Assessments

Within the context of usability assessments, simulation of real-world phenomena in

controlled laboratory environments represents a relatively novel approach. Some early

usability studies of mobile systems and services, in which key contextual features have been

replicated in laboratories, are described by Bohnenberger et al. (2002), Pirhonen et al. (2002),

and Kjeldskov and Skov (2003). However, there is little HCI literature describing how to

effectively compose such evaluation, and the mechanisms through which they can be

adjusted to fit a particular purpose, e.g., to inform specific design issues. This has motivated

us to search for guiding principles beyond what can be found in HCI and computer science

literature. In particular, we have studied literature from training simulation research. To

highlight similarities (and differences) between training simulations and usability

assessments we will continue by making a conceptual comparison of the two.

Simulation-based approaches have long since been systematically used for training purposes

in safety-critical industries. In the education of, e.g., aviation pilots and ambulatory care

personnel simulations form an essential part of the training (Patrick, 2002; Maynes, 2008).


10

For trained professionals, simulators are often used to maintain (or adapt) skills and practice

drills. The objective of training simulators is mainly to develop human work-related skills1.

This makes transfer of training, from simulations to real-world practice, a critical issue in

training design (Liu et al, 2008). Within research on training simulations there has been an

ongoing debate on the degree of realism, or fidelity, that need to be mirrored in simulations in

order to maximize transfer of training.

While training simulations aim to optimize human task performance, usability assessments

are conducted with the intention of measuring and, in the case of formative evaluation,

identifying ways to enhance the performance of a product. Similar to training simulations,

usability evaluations are specific with regard to participants, the objective of the participants,

and the context in which the activities are carried out.

Training simulations seek to maximize participants’ skill transfer from the simulated context

to the real-world context. Usability assessments typically seek to maximize design

knowledge relevant for future development of a product, by gathering data on product

acceptance and usability. Thus, the two types of exercises target different stakeholders. The

“learners” in usability assessments are not the test participants, but the evaluators.

Table 1 sums up the main difference between training simulations and usability evaluations.

In many ways, these differences make training simulations and usability evaluations

complementary when it comes to optimizing interaction with tools used as part of work-

related activities. Training simulations is about adapting people for specific tasks, while

formative usability evaluation is about adapting tools for specific tasks and users.

1 In certain cases, knowledge and attitudes may also be targeted through simulation (Beaubien & Baker, 2004). For the sake of simplicity, the current paper will only consider simulations used for skill training.


11

Training simulations Usability Evaluations

Purpose Enhance human skill

performance in a specific

context.

Evaluation of product

performance relative to a

particular context.

Learner/knowledge

recipient

Trainee. Evaluator.

Role of technology Part of simulator Product to be evaluated

Role of participant As trainee As representative user

Output Skill. Product acceptance and

usability.

TABLE 1 Comparison of training simulations and usability evaluations.

Current training simulation research strongly suggests that optimization of skill transfer is

intimately dependent on multiple factors including the type of skill being trained, who the

trainees are and their level of experience, the circumstances in which the training takes place,

and available resources (Liu et al., 2008). These different parameters typically guide how

training simulations are composed. Different simulation fidelity components are typically

adjusted to meet the objectives and premises of the training.

The position argued in the remainder of the current paper is that much of the same intimacy

between objectives and premises on the one side, and how training simulations are designed

on the other side, also applies to simulation-based usability assessments. Effective training

simulations are tailored to promote acquisition and transfer of specified skills in relation to

particular circumstances. We argue that usability assessments of mobile ICT for hospitals are

most effective in when they are tailored to promote end user reflection concerning specific

aspects of design. The underlying principle of simulations for both training and formative


12

usability assessment purposes is very similar. In both cases, the multitude of factors affecting

human cognition and behavior in the actual performance contexts makes it infeasible to

address them all at once, and understand their relevance and relationship.

Systematic decomposition of design problems is by no means a new concept in HCI and user-

centered design. Iterative design processes gradually increases the fidelity prototypes from

low to high. This is a central feature of the user–centered design process, as defined in ISO

13407 (1999). As pointed out earlier, this decomposition has typically only concerned the

prototype (typically the graphical user interface) and to a far less degree been applied to the

external use context.

5. Simulation Fidelity Theory

Having drawn some parallels and contrasts between training simulations and usability

assessments, we will now turn attention to aspects of simulation fidelity theory of particular

relevance when designing usability evaluations of ICT supporting mobile clinical work.

Drawing on relevant literature, this section will describe some central mechanisms or

components through which simulations for training purposes are modified to maximize their

effect.

5.1. Simulation Fidelity Components

As previously noted, fidelity in simulation training plays an important part in the transfer of

skills from exercise to reality. At an overall level, simulation fidelity defines the reality of the

simulation (Alessi, 1988; Gross, Pace, Harmoon, & Tucker, 1999). The exact understanding

of what “reality” consists of varies. Over the years, multiple fidelity dimensions have been

suggested (Rehmann, Mitman, & Reynolds, 1995). Some of these suggestions have been

highly simulator specific in the sense that they only apply to a given type of training device


13

for a specific domain. Motion fidelity in flight simulators, i.e., the extent to which the

simulator realistically mirrors in in-air movement, is an example of this. In the following

review, we have deliberately focused on fidelity dimension that are more universal, and that

can be used to describe simulations in a more general sense.

Simulation fidelity can be understood both in relation to the physical experience on the one

hand, and the psychological or cognitive experience on the other. This has motivated

different approaches to resolving the transfer problem. Physical fidelity and psychological

fidelity can be divided into subcomponents. Figure 1 presents an overview of the components

or dimensions of simulation fidelity that we will focus on in the current work.

FIGURE 1 Simulation fidelity dimensions.

5.1.1. Physical Fidelity (Engineering Fidelity)

Early transfer research emphasized the need for simulations to duplicate the physical

elements of their real-world counterparts – The more the training situation resemble the

characteristics of the real task situation, in terms of operational equipment and environment,

the more effective transfer of training can be expected. This hypothesis is also known as

identical elements theory (Thorndike & Woodworth, 1901). Equipment fidelity and

environment fidelity are often referred to as subcomponents of physical fidelity.


14

Equipment fidelity. Equipment fidelity refers to the extent to which the appearance and feel

of real tools, devices, or systems (hardware and software) that simulation participants operate

on is replicated (Zhang, 1993). For example, aircraft cockpit procedures have been trained

both with hi-fi representations of aircraft instruments and with lo-fi mock-ups (Prophet &

Boyd, 1970).

Environment fidelity. Environment fidelity concerns the extent to which physical

characteristics of the real-world environment (beyond the training equipment) are realistically

represented in the simulation (Kinkade & Wheaton, 1972; Beaubien & Baker, 2004). In the

case of flight simulators, this corresponds to visual, auditory, and motion stimulus. High-

fidelity aircraft simulators are full-size replica of cockpits that duplicate the operational

aircraft environment and motions to great detail (Rehmann, Mitman, & Reynolds, 1995). In

flight training environments of less fidelity (e.g., desktop evaluation environments), visual

and motional cues are often reduced or absent.

Within training simulation, physical reproduction of the actual performance environment has

traditionally been considered a prerequisite for maximum transfer. The underlying motivation

behind simulations prioritizing physical realism is that the physical similarity between

training and performance context will enhance the potential for skill transfer. High physical

fidelity, combined with training scenarios based on actual events, has been the main answer

to transfer-of-training in domains such as aviation and military.

While the physical fidelity approach has been proved effective for numerous tasks, its main

challenges is that it can be costly in terms of resources (Liu, Macchiarella, & Vincenzi,

2008), and that it potentially can increase the cognitive workload on participants and thereby

impede learning (Alessi, 1988).


15

5.1.2. Psychological Fidelity

The idea that physical resemblance between constructed situation and actual situation is

essential for training transfer was later questioned, when it was discovered that cognitively

oriented tasks could effectively be practiced using mock-ups to represent key elements of the

situation. Prophet and Boyd (1970) highlighted this in an early comparative study of aircraft

cockpit procedures trained in authentic environments and with low-fidelity representations of

aircraft instruments. Similar findings, indicating that high-fidelity environments and devices

are not always required for positive training transfer, is described in work by Hays et al.

(1992) and Duncan and Feterle (2000). The methods employed in these studies have put

emphasis on what within in training design is referred to as psychological fidelity.

Psychological fidelity concerns the degree to which the simulation captures key

psychological processes of the performance domain (Kaiser & Schroeder, 2003; Kozlowski

& DeShon, 2004). It affects the extent to which a trainee psychologically and cognitively

engages in the training situation (Kaiser & Schroeder, 2003), and is closely related to how

participant perceives the “plot” (actions, events, and changes in circumstances) of the

simulation, vis-à-vis the real-world situation. Human perception, attention, decision-making,

memory, and action are factors that may influence psychological fidelity (Patrick, 2002).

Similar to physical fidelity, psychological fidelity plays a key role in achieving positive

reactions among simulation participants.

By trigger the central cognitive mechanisms relevant for on-the-job performance, rather than

focusing on replicating the concrete performance environment, the psychological fidelity

approach represents an alternative technique for promoting engagement among participants.

High physical fidelity approaches, on the contrary, tries to implicitly capture the key

psychological processes trough realistic reproduction of the performance environment

(Kozlowski & DeShon, 2004).


16

The psychological fidelity perspective is an emerging approach within training design

(Kozlowski & DeShon, 2004). It is in many ways analogous to low-fidelity prototyping

approaches in HCI (Kuutti, Iacucci, & Iacucci, 2002; Svanæs & Seland, 2004).

Task fidelity and functional fidelity are often considered subcomponents of psychological

fidelity.

Task fidelity. Task fidelity describes the degree to which tasks involved in the actual

environment for a given domain is replicated in the simulation (Zhang, 1993; Roza, 2000;

Hughes & Rolek, 2003). This typically affects the extent to which participants experience the

simulation as operationally realistic. In high-risk domains this is often a critical component.

In full-mission flight simulations, for example, deviation from operational practice is

generally experienced as negative user acceptance cues and might elicit different pilot

behaviors during training vis-à-vis the real-world performance (Rehmann, Mitman, &

Reynolds, 1995). Developing scenarios that replicate task demands of the real-world system

is one of the techniques used for enhancing psychological fidelity in training simulations

(Beaubien & Baker, 2004). In training exercises, task are sometimes isolated to investigate

certain issues (Liu, Macchiarella, & Vincenzi, 2008).

Thomas (2003) suggests that task fidelity is often equated with the physical and functional

fidelity of the simulation. This tendency, it is argued, has biased technological advancement

in development of training simulation, while less focus has been placed of recreating

authentic task scenarios.

Functional fidelity. Functional fidelity describes the degree to which the simulation reacts

like “the real thing”, i.e., that it provides realistic responses on the tasks and actions executed

by the participant. This does not necessarily concern only the functional fidelity of the

training equipment per se. For example, flight simulators with high functional fidelity does


17

not only contain flight control devices that acts like real equipment, but also provide realistic

aircraft motion cues in response to interaction with those devices.

In order for simulation participants to learn the exact consequences of their actions, high

functionality is often required. A flight simulator, for example, must maneuver as a real

aircraft in order to for positive transfer of such training aspects.

Functional fidelity and task fidelity are both essential for the credibility of training, which is a

key premise for skill transfer.

5.2. Factors Affecting Simulation Fidelity Configuration

In the design of simulation training there are multiple factors influencing the overall fidelity

configuration. Below, we will present some of the key factors.

5.2.1. Goals of Training

A fundamental challenge related to training design is identifying a suitable level of

simulation fidelity without compromising positive transfer. As suggested in literature on

training design, this is intimately dependent on the goal of the training (Maran & Glavin,

2003; Beaubien & Baker, 2004).

In maximizing training transfer, physical fidelity and psychological fidelity can play

complementary roles (Kozlowski & DeShon, 2004). Studies have shown that when training

cognitively demanding tasks and procedures, high transfer can be achieved with simple

simulations (Prophet & Boyd, 1970). For such purposes, high-fidelity components can be

removed without having negative effect on transfer. This is especially the case if the

component is unimportant for the skill being trained (Maran & Glavin, 2003). For

inexperience subjects practicing basic task skills, advanced equipment and undesired


18

interference can be inappropriate and actually diminish transfer as it can divert trainees’

attention from the goal of the simulation.

5.2.2. Cost-to-Benefit

While high-fidelity simulations attempt to minimize the differences between training and

transfer situation, such approaches are often associated with high costs. This has made the

fidelity issue in training design a question of cost-to-benefit.

Drawing on Miller (1953), Alessi (1988) suggested that the degree of simulation fidelity

should ideally correspond to the training stage of the learner. Alessi argued that beyond a

certain level, increasing the fidelity of the training device would yield diminishing transfer.

Thus, increasing engineering fidelity beyond a certain level will not produce sufficient extra

transfer to make it worth the added costs. Kinkade and Wheaton (Kinkade & Wheaton, 1972)

proposed that the ideal relationship between physical fidelity and psychological fidelity

changes as the experience of the simulation participants increases.

By focusing on replicating the cognitive demands, rather than prioritizing engineering

fidelity, psychological fidelity approaches form a cost-effective alternative to simulations.

5.2.3. Other Factors

Hays and Singer (1989) identified multiple factors influencing the relationship between

simulation fidelity and transfer, including the abilities of instructors, instructional techniques,

types of simulators, measurement techniques, etc.

5.3. Summary of Key Simulation Fidelity Considerations

Based on the studies referenced in this section we have identified a number of fidelity

considerations that are relevant for training design. Below, we will first present a brief

summary of the key points from our literature review.


19

• The fidelity concept: At an overall level there are two major simulation fidelity

components – Physical fidelity and psychological fidelity. Physical fidelity is the

exactness to which real world operational equipment and environment are replicated

in the simulation. Psychological fidelity refers to the degree to which the central

cognitive processes relevant for the skill being trained are triggered. Each of the two

components can be further divided into subcomponents. Fidelity in modern training

design is considered a multi-dimensional feature.

• User acceptance: Reflecting a sufficient degree of fidelity is a necessary for the

trainees to accept the simulation as a replacement for the real world performance

context. This acceptance is required for provoking realistic behavior and positive

training transfer.

• Effectiveness: There is no direct correlation between the level of fidelity and

effectiveness in form of training transfer.

• Fidelity configuration: Fidelity is only critical in terms the role it plays in the

simulation experience. Simulation fidelity (both physical and psychological fidelity)

needs to be carefully adapted so that it matches the objectives and the content of the

training and training level of the participants.

• Iterative process: A training program will usually require different levels of

simulation fidelity as it progresses.

5.4. Relevance of Simulation Fidelity Considerations in the Design of Mobile

Usability Evaluations

If we compare the fidelity consideration above with the role of fidelity in usability

evaluations, there are some noteworthy parallels as well as distinctions.


20

• The fidelity concept: With regard to the general understanding of fidelity, as it is

described in HCI literature, it is to a much larger extent treated as a one-dimensional

feature, typically refereeing to how closely the graphical user interface prototype

matches the final product. The concept is normally not used to describe the accuracy

of a simulated use context.

• User acceptance: Positive transfer of training strongly depends on the users

acceptance of the simulation as a credible surrogate for the actual performance

context. Similar to training simulations, user acceptance is also critical in usability

evaluations, but for different reasons. A central issue in formative usability

evaluations is to provoke behavior and feedback for participants that can help inform

future design. This means that users need to be able to relate an evaluated design to

work-related tasks. If not, one cannot expect them to provide valuable feedback.

• Effectiveness: Effective training transfer is not a direct function of simulation fidelity.

Similarly, the quality of results from usability assessments does not per se depend on

whether a low-fidelity or high-fidelity prototyping approach was employed. It is

rather that different fidelity approaches in usability evaluations produce different

types of usability data. Low-fidelity prototype approaches typically put emphasis on

understanding the purpose of a computer system and context in which it will be used.

What are people trying to accomplish? Which processes are involved? What are the

essential user requirements? High-fidelity prototyping approaches, on the other hand,

are far more concrete with regard to the design solution being evaluated.

Consequently they tend to gather responses that to a much larger extent are solution

specific.

• Fidelity configuration: Simulation for training purposes attempts to tailor the fidelity

level of the trial to match the phase of the training program. Likewise, the level of


21

prototype fidelity in usability evaluations typically depends on which design phase it

relates to. Training simulations, however, has to a much larger extent than usability

evaluations systematically taken into account the performance context in which

control devices are used.

• Iterative process: Skill training and design of interactive systems are iterative

processes. In both cases the relevant performance context is typically to complex to

understand in a one-time effort. Instead, one first attempts to understand the basic

requirements, and then gradually addresses more complex issues.

6. Elements of the Use Context Affecting Mobile Usability in Hospitals

To empirically ground our previous claim that the usability of mobile ICT in hospitals is

determined by multiple factors that are highly contextual, we will present examples from two

previous simulation-based assessments of relevance.

The respective results from the two assessment are describe in detail in previous work

(Svanæs & Alsos, 2006; Dahl & Svanæs, 2008). In the current work, we will focus on aspects

of the two simulations that have not been discussed earlier – their fidelity compositions and

the compositions’ impact on the results.

6.1. Overall Experimental Design

Both assessments aimed to explore alternative interaction techniques supporting mobile

hospital workers at the clinical point of care (i.e., at the patient’s bedside). The assessments

were conducted in a large usability laboratory configured as a hospital ward section.

The main purpose of the experimental design for both assessments was to evaluate product

usability in a relevant context of use. According to ISO 9241-11, usability is always relative

to the context in which the product is used. In the standard, context of use is defined as


22

“users, tasks, equipment (hardware software and materials), and physical and social

environment in which a product is used.” Figure 2 provides a conceptual view of the various

components of the research setting, including the evaluated prototype, the simulated context

of use, and the usability laboratory with technical facilities (e.g., recording equipment).

.

FIGURE 2 Conceptual view of the experimental design.

6.2. Assessment 1: Combining Handheld Devices and Bedside Patient Terminals

In Assessment 1, we explored the potential for letting physicians use handheld devices

(PDAs) as input device for the bedside terminals. Eight different prototypes of this setup

were implemented, including a baseline solution where all interaction was done directly on

the patient terminal touch screen without using a PDA. The eight alternative designs were

tested in a pre-surgical scenario where a physician uses a bedside terminal to show X-ray

images to a patient. Figure 3 shows two of the design solutions. The illustrated solutions

allowed the physician to use select an X-ray image by dragging it to a terminal icon on the


23

PDA (left), or to use the PDA as a remote control to navigate in a menu on the bedside

terminal (right).

FIGURE 3 Design solutions for controlling patient terminals from a PDA.

The usability tests were done in a simulated use context replicating a patient room with a

hospital bed, a touch screen bedside terminal, and a PDA. A total of five pairs, one physician

and one simulated patient, were recruited for the tests. Figure 4 shows a physician interacting

with one of the design alternatives.

FIGURE 4. Test subject interacting with a PDA to select an X-ray image to show to

the patient.

After testing all versions, the physicians’ and patients’ opinions about the design solutions

were collected in a post-test interview and through a ranking exercise. The comments made

during the tests and in a post-test interview were analyzed. This gave insights into the factors

that were perceived as influencing the usability of the solutions.


24

6.2.1. Fidelity Configuration for Assessment 1

Regarding the fidelity configuration for Assessment 1, prototype (equipment) fidelity can be

characterized as medium. Prototypes were represented with real hardware, but only

simplified GUIs and information content (X-ray images) were employed. Environment

fidelity can be characterized as medium to high. It was configured using a real patient

bedroom a model, but certain physical elements were removed (e.g., patient uniforms,

blankets, paper charts attached to beds, medical equipment, etc.). We consider task fidelity to

be medium. A physician confirmed the realism and validity of the overall scenario in a pre-

test interview. However, we realize that low-fidelity information content may have affected

how the simulated pre-surgical consultation was carried out. Functional fidelity can be

categorized as being medium to high. The prototypes supported all the relevant interaction

techniques and provided feedback to users. In addition, the patient actors had been given a

brief instruction on how to behave during the assessments and what type of responses to

provide the test participants.

6.2.2. Summary of Identified Usability Factors in Assessment 1

Screen size and ergonomic aspects of devices. All participants reported that the patient

terminal screen was large enough for viewing X-ray images, while the PDA screen was too

small for this purpose.

Positioning the patient terminal within the range of the patient and the physician made it easy

for both to see the X-ray images. The terminal position made it easy for the patients to use,

while some physicians were uncomfortable with the solution, as they had to bend over the

patient’s bed to reach it. Some physicians commented that a good thing about the PDA-based

design alternatives vs. the baseline alternative was that they no longer had to bend over the

patient’s bed to interact with the terminal.


25

Shared view vs. hiding information. One recurring issue during the interviews was whether

the list of available X-ray images should be on the patient terminal or only on the PDA. Most

physicians wanted to hide it because it distracted the patient, wasted valuable screen space, or

displayed patient sensitive information. The patient actors, however, felt that the physician

was keeping secrets for them when the list was not present.

Focus shifts. Almost all physicians commented that the PDA became a disturbing element in

the communication with the patient and that the change of focus between the PDA and the

patient terminal was quite demanding. The participants preferred less demanding design

alternatives that did not disturb them.

When the physicians and the patients looked at or interacted with the same screen, they felt

that they were communicating on an equal “level”. When the physicians started using the

PDA, some of them expressed that it became a disturbing element in the dialog and that they

now were communicating on different “levels”.

6.3. Assessment 2: Automatic Identification of Patients at the Point of Care

The aim of Assessment 2 was to evaluate and compare the usability of different sensor-based

techniques for automatic patient identification during medication administration in a ward.

During the drug administration round, a health worker distributes prescribed medicine to

ward patients, and signs off on the respective patients’ medication chart after each

administration. For simplicity, the chosen test setup involved only two in-bed patients. It was

assumed that the correct medicine dosage for the respective patients was carried in the

hospital workers’ pockets.

The problem being addressed in the developed prototypes was that of identifying the correct

patient at the point of care. By adding new ubiquitous-computing technology to the mobile


26

EPR, such as token readers or location sensing, patient identification can be automated

thereby reducing potential for error during administration.

Four different design solutions were evaluated. The four alternatives were combinations of

two sensing technologies and two device technologies (Figure 5). The two sensor

technologies were barcodes (token-based) and WLAN positioning (location-based). The

employed positioning system continuously detected the physical position of all WLAN

devices in the simulated patient room to an accuracy of approximately 0.5 meter. PDAs

(mobile) and bedside touch-screen terminals (stationary) formed the two device technologies.

An implicit assumption in the prototype implementations was that the devices could retrieve

medication charts from an EPR system. The user interface for the medication chart was

extremely simplified, as the focus of the study was not on medication charts or the GUI, but

on automatic identification of patients.

FIGURE 5. Two of the four design solutions that were evaluated. The left image

shows location-based interaction with electronic patient charts being presented on a

mobile device. The right image shows a token-based variant with patient charts being

presented on a fixed bedside terminal.

Eight hospital workers from a local hospital participated in the assessment. Persons with

experience from health care acted as patients during the simulations. The test participants


27

were encouraged to interact with the simulated patients in a similar way as they would

interact with real patients in an authentic care situation.

As in Assessment 1, the participants were asked to rank the design solutions in order of

preference and to explain their priories. The transcripts from the ranking sessions were

examined to identify factors that influenced their ranking.

6.3.1. Fidelity Configuration for Assessment 2

The fidelity configuration for Assessment 2 can be summed up as follows. Prototype fidelity

can be described as being medium to low. Similar to Assessment 1, the simplified GUIs and

information content offered by the prototypes reduced their fidelity in spite of using real

computing devices as part of the test. Similar to the previous assessment, the environment

fidelity can be described as medium to high. The lack of physical representations of the

medicine to be administered as well as medication trays was the primary features that were

missing from the simulation vis-à-vis the performance context. We regard task fidelity as

being medium. At an overall level the simulation reflected the key tasks that are part of an

administration round (e.g., moving between patients, informing about medication to be

administered, signing), but details concerning the administration was deliberately omitted.

This, for example, included physically handing over medicine to patients. Similar to the case

for Assessment 1, we consider the functional fidelity of Assessment 2 to be relatively high.

The sensor-based interaction techniques were implemented using high-fidelity sensor

technology. As with the previous assessment, the patient actors had been given general

instructions on how to respond during the simulation.

6.3.2. Summary of Identified Usability Factors in Assessment 2

Attention on computer devices vs. attention on patient. Many test participants expressed a

general concern that cumbersome information navigation would require them to pay too


28

much attention to the computer devices, rather than attending the patient. They consequently

all saw the benefit of automatic patient identification.

The two location-based interaction techniques were ranked high. These design alternatives

made use of the clinician’s natural mobility in the physical environment. The fact that these

techniques allowed patient identification to occur in the background of the user’s attention

can be viewed as an important reason for their high rating.

In order to retrieve patient information via tokens (i.e., barcodes), the users had to explicitly

scan them. The test participants who preferred location-based interaction to token-based

interaction argued that barcode scanning took attention away from the patient and the care

situation.

Predictability and control. Earlier work on context-aware and ubiquitous computing has

suggested that autonomous computer behavior often implicates loss of user control (Bellotti

& Edwards, 2001; Barkhuus & Dey, 2003). The conducted usability tests revealed similar

findings. Users that preferred token-based interaction to location-based interaction found that

getting computer response as a result of an explicit and deliberate action gave them a feeling

of greater control over the application. This made the users more certain that they were

signing off on the correct patient medication chart.

We found that the potential lack of control some users experienced when testing the location-

based solutions was related to the fact that the zones in the room were invisible. Thus, the

physical test environment did not provide any visible cues to participants when to expect

system response based on their physical movements. Despite the lack of control many users

experienced with the location-based solution, many were willing to give up control as long as

it made patient identification easier.


29

Integration with work situation. Most test subjects commented that when administering

medicine in their everyday work, they were accustomed to informing the patient verbally

what medicine he or she was given. Many of the test participants therefore saw an additional

benefit of having the opportunity to visually show medical information to the patient via the

shared screen of the bedside terminal. Accomplishing this via the small screen on the PDA

was experienced as being far more cumbersome.

Several test participants expressed that another benefit of using stationary patient terminals

versus a portable device was that it allowed them to have both hands free. This was seen as

important as they often perform tasks at point of care that require both hands free (e.g. hand

over medicine, help patients in and out of their beds). Hence, most test subject found the

fixed bedside terminals to be more integrated with the overall work situation, while the PDA

imposed more of a physical constraint.

One of the potential drawbacks of the implementation involving a stationary device, as

pointed out by test participants, was related to privacy. When using a shared screen it is also

possible for others (e.g., patients and visitors) in the room to see the information.

6.4. Summary of Results

As the results presented above from suggest, the two assessments identified a number of

usability factors external to those of the GUI of the evaluated design solutions. Assessment 1

identified issues concerning screen size and ergonomics of patient terminals, sharing (and

hiding) of patient related information, and focus shifts between multiple devices during

patient visits. Assessment 2 identified issues related to time spent on computer devices vs.

time spent on patients, predictability and user control, and integration with physical and

social aspects of the work situation.


30

7. Analysis

While we regard the results above highly relevant in terms of informing future design of

mobile ICT for hospitals, they also raise some interesting implications for how we evaluate

the usability of such technology. In particular, we consider the issue of how the respective

fidelity configurations of the two assessments contributed to the findings to be relevant.

Essentially, the analysis provided below highlight the close relationship between fidelity

configuration of the simulation-based usability assessments and the type of feedback we

received from the participants.

7.1. Environment Fidelity

The custom-designed laboratory that the usability evaluations were conducted in provided a

test environment accurate with regard to the physical size and configuration of patient rooms

that the test subjects knew from their everyday work. The significance of replicating key

physical aspects, such as room proportions, patient beds, and in-bed patients, is particularly

evident if we consider the physical and bodily aspects of usability that were identified

through the experiments. An example of this is the findings concerning shared information

displays. In both assessments, participants found it highly valuable to share a common screen

display with in-bed patients. Participants commented this after they had been given concrete

ideas of the physical configuration of different design solutions, i.e., the physical placement

of the terminal in relation to the patient bed and in relation to the in-bed patient. In the setup

shown in Figure. 4 (p. 23), the patient actor in the bed is not only needed for facilitating a

patient-caregiver dialog during test scenario. Responses from participants also suggested that

the patient actors’ physical position and posture also helped give an idea of how suitable the

design solutions were for sharing information between caregiver and patient. In addition

these physical factors gave an idea about the ergonomic suitability. In many cases the


31

participating hospital workers used the patient actor in the bed and the terminal as physical

reference points in the room.

Another simple example highlighting how certain physical aspect of the environment can

help detect ergonomic requirements is shown in Figure 6. A recurring comment from the test

participants was that during point-of-care situations they could strongly benefit from having

ICT solutions that are physically ready at hand, and that can be brought forth or temporarily

stored in the background depending on what the immediate care situation calls for. During

the evaluations, test participants would occasionally put handheld terminals in the pocket of

their work uniform when they, e.g., needed to physically examine patient actors. The

handheld terminal would later be picked up to complete the simulated tasks. Thus, outfitting

test participant with actual hospital uniforms helped indicate that hospital workers can benefit

from mobile media having a physical size that easily fit their pockets.

FIGURE 6 In this case the work uniform the test subject wears helps identify a

relevant usability requirement – The handheld terminal has an acceptable size, fitting

the test pocket of the uniform, making both hands free for the clinician.

Although the examples above may seem trivial when considered in isolation, they serve to

illustrate the intimate relationship between the physical environment and the size and form


32

factors of interactive devices used at the point of care. They illustrate how physical cues in

the test environment can help them relate design solution to their everyday work. By being

concrete with regard to certain physical features (room proportions, furnishing, in-bed

patient, work uniforms) the test environment explicitly encouraged participant to think about

physical and bodily aspects of their work in relation to the solutions. At the same time, none

of the participants responded negatively to physical features (e.g., medical drugs and trays,

blankets on beds, patient uniforms) that were missing in the simulations, but normally be a

part of actual performance context.

We have not been able to find results from conventional usability test of applications running

PCs that identify similar physical and bodily usability factors. This suggests that participants

need to be exposed to a sufficiently realistic physical work environment during testing in

order for such usability issues to surface.

7.2. Prototype (Equipment) Fidelity

Both assessments involved the use of working prototypes. The fidelity of the prototype GUIs

were deliberately kept low, showing only simplified graphical representations of medical

information. This was essentially to promote feedback describing how users experienced

novel interaction techniques in relevant work situations, which was primary objective of the

evaluations. The behavior of the participants during simulations and comments in post-test

interviews suggested that they, despite simplified GUIs and medical content, perceived the

simulation experience as sufficiently realistic. For example, test participants frequently

evaluated the different candidate solutions against current (and often paper-based) practices,

and made usability related comments as if the simulation experience had occurred in the

actual performance context. Often, these comments would be related the form factors of

design solutions, and how they accommodated the care situation.


33

7.3. Task Fidelity

Task fidelity plays a central role in helping participants place the simulation in a context – In

our case, relate the simulation experience and design solution to their everyday work. The

tasks that the test participants were given were all related to particular clinical scenarios

(administration of medical drugs, pre-surgical briefings), but fairly open in the sense that they

did not strictly dictate how the participants performed the tasks. The main motivation for

providing the test participants fairly generic tasks was that we wanted to do an early

exploration of the design space for each of the cases, and that we did not have a precise

understanding of relevant usability factors. As such, we considered it important to allow for

central usability factors arise out of the way the test participants chose to solve the tasks, and

draw on their work experience. We also recognized that providing more detailed tasks would

possibly require higher GUI prototype fidelity and more accurate medical information

content.

Based on our experience, the tasks play an important role in providing participants credible

reasons for executing behavior of relevance during usability assessment. For example, the

tasks made sure that the participant would communicate with the simulated patients and

move between different beds, and interact at least once with each candidate solution during

the evaluation.

7.4. Functional Fidelity

As previously noted, we considered the functional fidelity of the conducted assessment to be

fairly high. By employing high-fidelity sensors and interactive devices in the tests, the

participants where arguably given a relatively realistic experience of how the different design

concepts could work in an actual point-of-care context with state-of-the art technology. In

particular, this was relevant with regard to the findings concerning predictability and control.


34

We consider it unlikely that test participants would have reflected on these issues without

being given a realistic impression of how the design concepts present themselves to users in

practice.

The patient actors also played a part in providing realistic responses to the tasks and actions

executed by participants. Among, the patient actors most important functions was to help the

test scenarios progress by responding to information from and questions asked by the

participating clinicians. In addition, they served as the primary focus of attention for the

clinicians during the simulations.

7.5. Simulation Acceptance Model

We have applied simulation-based usability assessments to evaluate early prototypes of

mobile ICT for hospitals. In order to provoke realistic behavior and design relevant

reflections among hospital workers during simulated care scenarios, however, they need to

accept the simulation experience as a credible replacement for real-world episodes.

Simulation fidelity plays a key role in this context. Drawing on the analysis above, Figure 7

proposes a simulation acceptance model. The figure gives a conceptual view of the factors

that influence the extent to which a simulation experience evokes commitment and

engagement among participants. The overall simulation fidelity configuration affects how

each participant perceives the realism of the simulation experience. Evaluators may adjust the

various simulation fidelity dimensions according to the goal of the assessment. The required

accuracy of each fidelity dimension, however, is ultimately dependant on the individual

participant. It is essential that all fidelity dimensions exceeds or equals a lower threshold (f)

for him or her. Failure to do so is likely to cause a negative user experience, and make it

difficult for the participant to reflect on the usability of a prototype. However, it can be

argued that high-fidelity components may partly compensate for low-fidelity components if


35

they interact closely (Schricker, Franceschini, & Johnson, 2001). If all fidelity dimensions

meet the requirements of a specific test subject (∏), simulation acceptance is achieved.

Simulation acceptance is one of the key factors affecting the engagement and commitment of

participants during a trial. Other influencing factors include personal interests, attitudes

towards technology and approach, rewards for participating, etc.

FIGURE 7 Simulation acceptance model.

8. Discussion

In the current section we will briefly discuss some issues of relevance to simulation-based

usability assessments of mobile ICT for hospitals.

8.1. Replicating a Sufficient Degree of Realism?

Rather than incorporating highest possible fidelity across multiple dimensions, the analysis

above suggest that a more feasible approach is to carefully select which aspects of the

performance context to replicate accurately, and which aspects to simplify or remove. The

motivation behind customizing the simulated use context this way is that it allows evaluators

to gradually increase the complexity of the simulated use context as the product is developed.


36

A fundamental question that arises from such an approach is: How can you tell design-time if

you are incorporating a suitable degree of realism into a simulation-based usability

assessment, so that relevant user reflections are evoked? We have suggested various aspects

of the research setting (fidelity components) that may affect the perceived realism of a

simulation-based usability assessment, and that these have to be adjusted according to what

issues that one wants to put focus on. Yet, we recognize that there is no definite way of

telling a priori whether the gap between the simulated context and the performance context is

successfully bridged or not. Having a simulator (i.e., a customized usability laboratory)

allowing for realistic mobility and care situations to be acted out in a physically realistic

environment is not a “valid” system for evaluating mobile usability for hospitals per se.

Rather, validity of usability data is dependant on the focus of the evaluation and the context

from which the data was derived. Below, we describe some applicable techniques and

measures that can be taken to help ensure that a simulated use context can generate valid

usability data.

8.1.1. Using Consultants from Health Care

During preparations of simulation-based usability assessments we have used hospital workers

as consultants. Nurses and physicians have assisted in identifying clinical situation that could

benefit from mobile ICT. Moreover, they have participated in pilot tests to help verify that

the structure of the simulated task scenarios is in accordance with real hospital practice.

8.1.2. Field Studies

One of the main motivations for employing simulation-based usability assessment has been

to overcome practical and ethical challenges of studying mobile usability in the field. This is

not to say that simulations eliminate the need for field studies. On the contrary, field studies

described, e.g., by, Bardram and Bossen (2005), Munkvold and Divitini (2006), and Sørby et

al. (2006), give a rich picture of the dynamic nature of clinical work providing insights to


37

how information tools are used, and how hospital workers constantly face interruptions and

changes tasks. Thus, field studies provide knowledge that are essential for understanding the

underlying processes of clinical routines. This is critical for designing operationally realistic

simulations. We also recognize the need for post-simulation field studies to help validate

results. In this sense, we consider simulation-based usability assessments and filed studies

complementary approaches.

8.1.3. Learning Process

Learning to use simulation-based usability assessments as an effective tool in the design of

mobile ICT for hospitals is a continuous process. Factors that typically affect fidelity

configuration include the goal of the assessment (i.e., which design aspects are being

addressed), cost-to-benefit of increased fidelity, and how far the evaluated concept is from

becoming a finalized product. Balancing these criteria requires experience.

8.2. Limitations of the Current Study

The current study is retrospective in nature. We have analyzed data from former usability

evaluations of mobile ICT for hospitals, using simulation fidelity theory to identify how

various aspects of the research setting contributed to the findings. This stands in contrast to

developing a theoretical basis concurrently as studies are carried out. Weaknesses of

retrospective studies include possible selection bias towards examples from trials that

confirms, rather than rejects, a theory.

We also recognize that the lack of systematic evaluations, in which different simulation

fidelity configurations are applied to the same test case, makes it impossible to compare and

estimate the impact different setups have on identified usability issues.


38

9. Summary and Concluding Remarks

9.1. Summary

In the current work we have discussed the relevance of simulation fidelity theory in the

design of assessments addressing the usability of mobile ICT for hospitals. In training

simulations design, fidelity is considered a multi-dimensional concept. Various features of

simulations are typically adjusted to meet specific training goals. By systematically adjusting

fidelity of various features, simulations can form effective training tools helping participants

to focus on relevant issues.

We have argued that the principle of carefully adapting the simulated context is highly

applicable for usability evaluations in order for them effectively inform design. This is

particularly relevant when evaluating mobile ICT for complex use settings, such as hospitals.

We supported our argument by drawing on result from two relevant usability assessments.

The assessment indicated that relevant usability factors in mobile clinical care situations

often are related to aspects of the external use context. We then how the overall fidelity

configuration for the two assessments helped evoke relevant behavior and reflections among

test participants. Based on the review we proposed a simulation acceptance model. The

model describes key features affecting the extent to which simulation-based usability

assessments evoke commitment and reflection among test participants.

9.2. Concluding remarks

This paper has presented an alternative view to the fidelity concept compared to the common

understanding of the term in HCI. Drawing on simulation research we have proposed that the

concept can also be used to describe the accuracy of various features in simulated use context

beyond the evaluated products. The need to evaluate usability of ICT in realistic context of

use is acknowledged in HCI and in international standards. Applying simulation fidelity


39

theory in the design of usability evaluation targeting complex use contexts, gives a more

nuanced view on the level of realism required in order for participant to accept simulated

trials. Rather than aiming for high fidelity across multiple dimensions, a more feasible

approach is do be selective about which dimensions of the use context to faithfully replicate.

By providing a sufficient degree of realism in usability assessments, as opposed to testing in

fully authentic settings, evaluators can systematically address how various elements of the

use context affect usability.

Evoking relevant user behavior and feedback during simulation-based usability assessments

is essentially about setting the stage (i.e., the research setting) right, and making participants

accept the illusion of the simulation. Setting the stage right is intimately dependent on the

objective of the assessment. If the objective is changed, the stage needs to be adjusted

accordingly.

Simulation fidelity theory has helped us become aware of the strong relationship between the

fidelity configuration of simulation-based usability assessments and the type of usability

factors that are identified through such trials. We encourage further research on this topic.

References

Alessi, S. M. (1988). Fidelity in the Design of Instructional Simulations. Journal of Computer-Based Instruction 15(2): 40-47.

Bardram, J. E. & Bossen, C. (2005). Mobility Work: The Spatial Dimension of Collaboration at a Hospital. Computer Supported Cooperative Work 14(2): 131-160.

Barkhuus, L. & Dey, A. (2003). Is context-aware computing taking control away from the user? Three levels of interactivity examined. Proceedings of Ubicomp 2003: 159-166.

Beaubien, J. M. & Baker, D. P. (2004). The use of simulation for training teamwork skills in health care: how low can you go? Qual Saf Health Care 13(51-56).

Bellotti, V. & Edwards, K. (2001). Intelligibility and accountability: Human considerations in context-aware systems. Human-Computer Interaction 16(2, 3 & 4): 193-212.

Bohnenberger, T., Jameson, A., Kruger, A., & Butz, A. (2002). Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach. Mobile HCI 2002, London, UK, Springer Verlag.


40

Dahl, Y. & Svanæs, D. (2008). A comparison of location and token-based interaction techniques for point-of-care access to medical information. Personal and Ubiquitous Computing 12(6): 459-478.

Duncan, J. C. & Feterle, L. C. (2000). The use of personal computer-based aviation training devices to teach aircrew decision-making, teamwork, and resource management. Proceedings of the IEEE 2000 National Aerospace and Electronic Conference, Dayton, OH.

Gagné, R. M. (1962). Simulators. Training Research and Education. Glaser, R. New York, Wiley.

Gross, D. C., Pace, D., Harmoon, S., & Tucker, W. (1999). Why fidelity? Proceedings of the Spring 1999 Simulation Interoperability Workshop.

Hays, R. T., J.W., J., Prince, C., & Salas, E. (1992). Flight simulator training effectiveness: a meta-analysis. Mil. Psychol. 4(2): 63-74.

Hays, R. T. & Singer, M. J. (1989). Simulation fidelity in training system design: Bridging the gap between reality and training Springer-Verlag.

Hughes, T. & Rolek, E. (2003). Fidelity and validity: issues of human behavioral representation requirements development. Proceedings of the 2003 Winter Simulation Conference, New Orleans, LA.

ISO 9241-11 (1998). Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) -- Part 11: Guidance on Usability.

ISO 13407 (1999). Human-centred design processes for interactive systems. Kaikkonen, A., Keklinen, A., Cankar, M., Kallio, T., & Kankainen, A. (2005). Usability

testing of mobile applications: A comparison between laboratory and field testing. Journal of Usability Studies 1(1): 4-17.

Kaiser, M. K. & Schroeder, J. A. (2003). Flights of fancy: the art and science of flight simulation. Principles and Practice of Aviation Psychology. Vidulich, M. A. and Tsang, P. S. Mahwah, NJ, Lawrence Erlbaum Associates: 435-471.

Kaptelinin, V. (1996). Activity Theory: Implications for Human-Computer Interaction. Context and Consciousness: Activity Theory and Human-Computer Interaction. Nardi, B. A. Cambridge, MA, USA, Massachusetts Institute of Technology: 103-116.

Kinkade, R. & Wheaton, G. (1972). Training device design. Human engineering guide to equipment design. Van Cott, H. and Kinkade, R. Washington, DC, Department of Defense: 668-699.

Kjeldskov, J. & Skov, M. B. (2003). Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System. Interact 2003, IOS Press, Amsterdam.

Kjeldskov, J. & Skov, M. B. (2007). Studying Usability In Sitro: Simulating Real World Phenomena in Controlled Environments. International Journal of Human-Computer Interaction 22(1): 7-36.

Kjeldskov, J., Skov, M. B., Als, B. S., & Høegh, R. T. (2004). Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Mobile HCI 2004.

Kozlowski, S. W. J. & DeShon, R. P. (2004). A psychological fidelity approach to simulation-based training: theory, research and principles. Scaled Worlds: Development, Validation and Applications. Schiflett, S. G., Elliott, L. R., Salas, E. and Coovert, M. D., Ashgate Publishing.

Kuutti, K., Iacucci, G., & Iacucci, C. (2002). Acting to Know: Improving Creativity in the Design of Mobile Services by Using Performances. Proceedings of the 4th conference on Creativity & cognition, Loughborough, UK, ACM, New York, NY, USA.


41

Liu, D., Macchiarella, N. D., & Vincenzi, D. A. (2008). Simulation Fidelity. Human Factors in Simulation and Training Vincenzi, D. A., Wise, J. A., Mouloua, M. and Hancock, P. A., CRC Press.

Maran, N. J. & Glavin, R. J. (2003). Low- to high-fidelity simulation – a continuum of medical education? Medical Education 37 (Suppl. 1): 22-28.

Maynes, R. (2008). Human patient simulation in ambulatory care nursing. AAACN Viewpoint 30(1).

McGrath, J. E. (1995). Methodology Matters: Doing Research in the Behavioral and Social Sciences. Readings in Human-Computer Interaction: An Interdisciplinary Approach. In Baecker, R. B., W. A. S. San Mateo, CA, Morgan Kaufman Publishers.

Miller, R. (1953). Handbook on training and training equipment design. Munkvold, G. & Divitini, M. (2006). From storytelling to reporting - converted narratives. In

proceedings of MCIS’06, the Mediterranean Conference on Information Systems, Venice, Italy.

Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J., & Stenild, S. (2006). It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. NordiCHI 2006, Oslo, Norway, ACM, New York, NY, USA.

Patrick, J. (2002). Training. Principles and Practice of Aviation Psychology. Tsang, P. S. and Vidulich, M. A., CRC Press.

Pirhonen, A., Brewster, S., & Holguin, C. (2002). Gestural and Audio Metaphors as a Means of Control for Mobile Devices. Proceedings of the SIGCHI conference on Human factors in computing systems.

Prophet, W. W. & Boyd, H. A. (1970). Device-Task Fidelity and Transfer of Training: Aircraft Cockpit Procedures Training. Tech. Report. Human Resources Research Organization, Alexandria, VA.

Reddy, M., Dourish, P., & Pratt, W. (2006). Temporality in Medical Work: Time also matters. Computer-Supported Cooperative Work 15(1): 29-53.

Rehmann, A., Mitman, R., & Reynolds, M. (1995). A handbook of flight simulation fidelity requirements for human factors research. Tech. Report No. DOT/FAA/CT-TN95/46. Wright-Patterson AFB, OH: Crew Systems Ergonomics Information Analysis Cente. .

Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., & Toscos, T. (2007). Why It’s Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. UbiComp 2007, LNCS 4717. Krumm, J. e. a., Springer-Verlag Berlin/Heidelberg: 336–353.

Roza, M. (2000). Fidelity considerations for civil aviation distributed simulations. Proceedings of the AIAA Modeling and Simulation Technologies Conference and Exhibit, Denver, CO.

Schricker, B., Franceschini, R., & Johnson, T. (2001). Fidelity evaluation framework. Proceedings of the IEEE 34th Annual Simulation Symposium, Seattle, WA.

Svanæs, D. & Alsos, O. A. (2006). Interaction Techniques for Using Handhelds and PCs Together in a Clinical Setting. NordiCHI 2006, Oslo, Norway.

Svanæs, D. & Seland, G. (2004). Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems. Proceedings of the SIGCHI conference on Human factors in computing systems, Vienna, Austria, ACM.

Sørby, I. D., Melby, L., & Nytrø, Ø. (2006). Characterising cooperation in the ward: framework for producing requirements to mobile electronic healthcare records. International Journal of Healthcare Technology and Management 7(6): 506-521.

Thomas, M. J. W. (2003). Operational Fidelity in Simulation-Based Training: The Use of Data from Threat and Error


42

Management Analysis in Instructional Systems Design. In Proceedings of SimTecT2003: Simulation Conference.

Thorndike, E. L. & Woodworth, R. S. (1901). The influence of improvement in one mental function upon the effciency of other functions. Psychological Review 9: 374-382.

Virzi, R. A. (1989). What can you learn from a low-fidelity prototype? In Proceedings of the Human Factors Society 33rd Annual Meeting., Santa Monica, CA.

Virzi, R. A., Sokolov, J. L., & Karis, D. (1996). Usability problem identification using both low- and high-fidelity prototypes. Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, Vancouver, British Columbia, Canada, ACM, Neew York, NY, USA.

Zhang, B. (1993). How to consider simulation fidelity and validity for an engineering simulator. Am. Inst. of Aeronaut. Astronaut.: 298-305.

Documents

Fidelity Considerations for Simulation-Based Usability …folk.ntnu.no/oleanda/papers/Fidelity considerations... · 2009-10-03 · Fidelity in Simulation-Based Usability Assessments