Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
JUST-IN-TIME SUPPORT: ADAPTIVE, INTELLIGENT SYSTEMS
TO ENHANCE HUMAN PERFORMANCE
by
Paul Picciano
A dissertation submitted to the faculty of
The University of Utah
in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
Department of Psychology
The University of Utah
May 2006
INTRODUCTION .......................................................................................................................5
What is Just-in-Time Support (JITS).......................................................................................5 The Growing Need for JITS ....................................................................................................5 JITS applied to Emergency Response: CPR and defibrillation ...............................................9
Sudden Cardiac Arrest (SCA)..............................................................................................9 Responding to SCA............................................................................................................12 JITS to Improve SCA Response ........................................................................................15
BACKGROUND .......................................................................................................................17
Intelligent Tutoring Systems (ITS) ............................................................................................18 Elementary Components of ITS........................................................................................19
Differences Between ITS and JITS........................................................................................21 Objectives ..........................................................................................................................21 Time available....................................................................................................................22 Processing strategies ..........................................................................................................23 System input.......................................................................................................................23
Foundations of JITS...............................................................................................................23 The Contextual Control Model (COCOM)........................................................................23 Expertise ............................................................................................................................29 Feedback ............................................................................................................................33 Visual Displays ..................................................................................................................36 Task Analysis (TA)............................................................................................................39
Structure of Just-in-Time Support Systems ...........................................................................43 Plans...................................................................................................................................44 Cues....................................................................................................................................45 Feedback ............................................................................................................................46
Hypotheses.............................................................................................................................48 METHODS ................................................................................................................................50
Participants.............................................................................................................................50 Training..................................................................................................................................51
Participant Training ...........................................................................................................51 Coder Training ...................................................................................................................52
Apparatus ...............................................................................................................................53 Procedure ...............................................................................................................................56
RESULTS ..................................................................................................................................59
Physiological Data .................................................................................................................59 COCOM Data ........................................................................................................................66
Control Mode.....................................................................................................................66 Protocol Adherence................................................................................................................68 Survey Data............................................................................................................................71
Action.................................................................................................................................72 Feedback ............................................................................................................................73 Outcome.............................................................................................................................75
DISCUSSION............................................................................................................................76
CPR Task performance ..........................................................................................................76 JITS Effect .............................................................................................................................78 COCOM Parameters as Dependent Measures .......................................................................81 Conclusions............................................................................................................................86
Limitations .........................................................................................................................86 Contribution of JITS ..........................................................................................................89
APPENDICIES ..........................................................................................................................91
Appendix A: BLS Flow Chart ...............................................................................................91 Appendix B : COCOM Coding Sheet....................................................................................94 Appendix C : Experimental Data Forms................................................................................95 Appendix D : Screening Questionnaire .................................................................................99 Appendix E: Instructions ....................................................................................................100 Appendix F : Statistical Calculations...................................................................................102
Fisher’s Exact Test: GRE-NO n = 6.................................................................................102 MANOVA for Sensor Data ............................................................................................103 Inter-rater Reliability .......................................................................................................110 MANOVA for COCOM Coding .....................................................................................115 MANOVA for Post-Run Questionnaire (Survey) Results...............................................118 Fisher’s Exact Test: Instances of Feedback ....................................................................121 Fisher’s Exact Test: Protocol Sequencing. .....................................................................122
Appendix G: CPR Release/Consent.....................................................................................123 Appendix H: Basic Life Support (BLS)...............................................................................124 Appendix I: The Future of Just-in-Time Support Systems.................................................125
REFERENCES ........................................................................................................................129
List of Tables
1. Group means for CPR performance variables. ...................................................... 61 2. Group means for COCOM classification (in percent). ........................................... 66 3. Mean number of protocol steps executed correctly (9 max).................................. 70 4. Number of responders exhibiting correct sequencing of subtasks......................... 71 5. "Action" survey results .......................................................................................... 72 6. "Feedback" survey results...................................................................................... 73 7. Number of participants provided feedback by the system.................................... 74 8. "Outcome" survey results....................................................................................... 75
List of Figures
Figure 1. 3 (Training) x 2 (Device) nested factorial design............................................ 52 Figure 2. The device in use on training mannequin......................................................... 54 Figure 3. Components of the device. ............................................................................... 56 Figure 4. Snapshot of video instruction "place headrest" ................................................ 58 Figure 5. Group means of chest compression rate for each cycle ................................... 64 Figure 6. Group means of inspired volume per breath for each cycle............................. 65
INTRODUCTION
������������������������������������������������������ ����������������� ����������������� ����������������� �������������������The 1999 film The Matrix portrayed examples of “instant expertise” by means of
downloading knowledge and abilities directly into the brain with plug-and-play
convenience. This technique proved particularly effective when surprise encounters
demanded unanticipated skills sets (such as the sudden need to pilot a rotary aircraft).
Here, a novice was thrust into a critical situation lacking the expertise to deliver a
successful (or even non-fatal) outcome. While the prospect of instantly imparting piloting
skills remains science fiction, the work presented here embodies a pursuit to design
support systems enabling operators to far exceed their baseline capabilities at the moment
an event demands intervention.
Just-in-Time Support (JITS) provides a theoretical framework designed to
enhance human performance employing contextually aware systems. Specifically, the
current focus of JITS systems aims to assist non-expert operators complete exigent tasks
via adaptive support. The development of JITS systems challenges designers to assess
task requirements, provide resources, and plan for contingencies. A robust understanding
of the users and their needs is vital for driving successful human-system interactions.
Most determinant, decomposable tasks are potential candidates for JITS. It is envisioned
JITS can provide significant benefits when confronting emergency situations,
infrequent/off-nominal procedures, and tasks novel to the user.
��������������������������������������������������������������������������������������������������������
Just-in-time support (JITS) systems are intended to bridge the expertise gap and
empower non-expert users with the ability to achieve favorable results in unfamiliar
situations. Human operators often struggle with tasks due to inadequate skills and
insufficient knowledge. Frustration mounts as novel encounters with technology
continue to proliferate burdening luddites as well as the technically savvy (Gleick, 1999;
Naki�enovi� & Grübler, 1991; Sheridan, 2002).
Astute observers understand the unceasing advancement and infusion of
technology is replete with benefits as well as challenges. Improvements in technology
deliver new capabilities, liberating efficiencies and welcomed convenience never before
experienced. Minimally invasive surgery, remotely piloted vehicles, and self check-out
lines have evolved largely through technological enhancements.
Unintended consequences also emerge, however. Capricious system behavior,
substantial error rates, and user frustration reflect unmitigated interaction deficiencies.
Users often find new technology enigmatic. The technology itself often receives the
focus without consideration for operator interaction. New products and systems continue
to demand more from operators (in terms of expertise) and provide less (interaction cues
and guidance). Thus, the number of human-system interactions involving novice
operators is expected to increase, resulting in unsatisfactory performance or an inability
to interact at all.
Unfamiliar tasks and technology confront society on a regular basis. This
challenge is omnipresent in developed modern nations. Consumers with new mobile
phones often cannot access their voicemail for days and encounters with the latest
features prove frustrating. Evolving home video and audio capabilities, with dozens of
functions hidden in an arsenal of remotes, frequently challenge users in the simplest tasks
(e.g., power on).
Even more pressing than a phone message, an unprepared public may encounter
urgent circumstances such as vehicle malfunctions, fires in their home, or the need to
administer life-saving treatment. A JITS system could provide assistance in all of these
situations by transferring expertise at the instant it is needed.
Current remedies addressing gaps in expertise have numerous limitations. Of
course, there are many training programs and simulations for teaching, and the use of
computers for learning is not revolutionary. However, current solutions for real-time,
adaptive support are scant. First and foremost, they fail to deliver the training
concurrently with the task. This hobbles the user with a significant memory burden that
is subject to temporal displacement (the time from training), and is further exacerbated by
the stress of the situation. For example, many large corporations utilize computer-based
training (CBT) that covers topics from intellectual property to evacuation. A CBT
approach helps teach large numbers of employees (it’s difficult to get 50,000 through
lectures), on their own time. However, there is difficulty retaining the information over
time, especially when employees view the training as peripheral to their job and simply
try to complete it as quickly as possible.
Second, the implemented expert models are often inflexible. Computerized
provider order entry (CPOE) systems provide such an example. Though designed to help
reduce adverse drug events and control costs, inflexible thresholds for drug dose and drug
interactions as well as pop-up windows have been a source of frustration for users
(Miller, R., Waitman, L., Sutin Chen, S. & Rosenbloom, S., 2005)
Further, these systems require direct manipulation (usually on a computer), which
may be unrelated to the goal. This obligates the operator to perform two duties - the
primary task (accomplishing the objective), and the secondary task of updating the aiding
system. One such scenario would be the case of an in-vehicle GPS directing a driver
down a closed road. Deficiencies such as described above inspired the development of
the JITS approach.
JITS solutions strive to circumvent task impediments and provide concurrent
support to novices in specific task objectives. By providing workable plans, directive
cues, and actionable feedback in a manner suited to the non-expert, a JITS system can
help novice users accomplish tasks that would otherwise have an extremely low
probability for success. The JITS framework success resides in providing users real-time,
adaptable, transparent, information to support task facilitation in the given context.
For instance, an untrained person faced with a victim requiring CPR may feel
paralyzed by the lack of knowledge and the urgency of the situation. For those willing to
make a magnanimous attempt, aspiration of the stomach and broken ribs could result
from improper CPR technique; this commonly occurs even when delivered by trained
professionals (Lederer et. al. 2004).
Such a circumstance provided an ideal proving ground for JITS as it encapsulated
all the parameters and challenges JITS was designed to mitigate. The demands of basic
life support (BLS, see Appendix H) are well defined and prescribed, yet demands
adaptation based on victim needs. The CPR task requires knowledge that is not
commonly held and skills widely unpracticed. The automated external defibrillator
(AED) compels an interaction with unfamiliar technology. The circumstance demands
immediate action in the absence of experts.
The nature of JITS is well suited for developing a BLS aid. Cues can initiate user
actions. The active sensors determine the adequacy of those actions and delivers
feedback to drive performance in the right direction. It was believed this approach would
be far more successful than a “canned” program that doled out instructions assuming a
putative response scenario without regard to the specific context. Current AED solutions,
such as those of Phillips and Zoll offer only a single treatment path which introduce
delays and may not be optimal for the victim.
The first step (after opening the package and removing the victim’s shirt) of the
Phillips HeartStart OnSite guides the responder through pad placement for heart rhythm
analysis (demo: http://www.medical.philips.com/main/products/resuscitation/products/onsite/onsite_demo.asp).
In contrast, Zoll’s AEDPlus (demo: http://www.zoll.com/parent2.swf) first instructs the
responder to call 911 and check for responsiveness. The debate here is not which
sequence is better, but to point out that neither system can adapt its sequence to best serve
the specific context. In some cases victim benefits from an immediate shock. Others
need oxygen delivered as soon as possible. The objective of a JITS system is to make
that determination and guide the user accordingly.
����������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������
��������������� ��������
Cardiovascular disease is the leading cause of death in the United States with over
696, 000 cases recorded by the Center for Disease Control (CDC) in 2002 (National
Center for Health Statistics, 2002). In particular, SCA claims 300,000 to 400,000
American lives annually (AHA, 2005, Huikuri, Castellanos & Myerburd, 2001, Zipes &
Wellens, 1998). Ventricular fibrillation is believed to be the insidious perpetrator
associated with the majority of these heart maladies (AHA, 2005, Cummins & Hazinski,
2000). In addition to the mortality in the United States, SCA causes 700,000 European
deaths per annum (ILCOR, 2005).
Unfortunately, most victims that experience SCA have only a small chance of
survival (Stiell, Nichol, Wells, De Maio, Nesbitt, Blackburn, & Spaite, 2003; Zipes &
Wellens, 1998). Increasing the probability of survival requires skilled and
knowledgeable responders. Usually the task falls to EMTs, firefighters, police officers,
or other highly trained responders. Unfortunately, skilled response alone does not
guarantee healthy outcomes; the elapsed time between incident and intervention is
believed to be more critical (Cummins, 1989, ILCOR, 2005 ). Disturbingly, there is a
growing literature demonstrating professionals’ deficiencies in both performance and
response time.
Despite their extensive training, studies continue to document trained responders’
suboptimal CPR performance. The accrued evidence suggests professionals must
improve their technique to propagate healthier outcomes (Abella, Avarado, Myklebust, et
al. 2005; Wik, Kramer-Johansen, Mykleburst , Sorebo, Svensson, Fellows, & Steen,
2005).
Abella et al. (2005) recently published their findings assessing the quality of CPR
delivered by professionals inside hospitals in Chicago between 2002 and 2004. They
observed numerous instances in which resuscitation performance did not adhere to the
American Heart Association (AHA) guidelines. The deficiencies included chest
compression rates below 90 per minute (the guideline prescribes 100/min), insufficient
compression depth, and excessive ventilation rates. The suboptimal performances
witnessed lead the researchers to conclude that even professional responders could
benefit from active monitoring and feedback during the task.
Wik et al., (2005) performed a similar examination but studied professionals out
of the hospital performing CPR in the field. This European study composed mostly of
paramedics and nurse anesthetists reported results consonant with Abella et al. (2005).
Of the total time available during the case, responders failed to provide chest
compressions for 48% of the available time. Valenzuela, Kern et al. (2005) indict the
frequent interruption of chest compressions for the poor survival rate in out-of-hospital
SCA. Insufficient compression activity combined with inadequate compression depth
(only 28% of compressions given adhered to the guideline) revealed a problematic
departure from the standards or CPR.
But even ideal execution of the protocol is futile if it is not delivered immediately
after SCA onset. Numerous studies assert that timeliness of response is the primary
factor in promoting healthy outcomes (Bunch et. al, 2003, Cummins & Hazinski, 2000,
Hallstrom & Ornato, 2004, Marenco, Wang, Link, Homoud, & Estes, 2001; Valenzuela,
Roe, Nichol, Clark, Spatie & Hardman, 2004). Each passing moment devoid of
intervention imperils the victim’s survival. For each minute without life-sustaining
measures, the victim’s probability of survival decreases 7-10% (Cummins, 1989).
Twelve minutes from collapse without circulating oxygen reduces the chance of survival
to a grim range of 2-5% (Cummins & Hazinski, 2000).
Survival rates attest to the historically slow response times. Zipes & Wellens
(1998) postulate a 3-5% discharge survival rate (victims that survive from cardiac event
to hospital discharge) nationally. In cities where there is widespread deployment of
AEDs and civilian training initiatives, survival rates make a small climb to a
disappointing 15%. Other studies lend credence to these concerns. Stiell, Nichol, Wells,
De Maio, Nesbitt, Blackburn, & Spaite (2003) reported on a 20 community study with
more than 8000 victims and found survival to discharge improved to a meager 5.2% with
citizen initiated CPR. Formerly a community with a superior discharge survival rate,
Dade County fell from 23% in the 1970s to only 9% in 1996 (Myerburg, Fenster, Velez,
Rosenberg, Lai, SKurlansky, Newton, Knox, & Castellanos, 2002). The alarming
decline served as the impetus for their examination of response times.
Significantly better survival rates have been observed in airline and casino
studies. These circumstances represent the confluence of optimized response
characteristics: trained, equipped rescuers in close proximity to provide immediate
resuscitation. An airline study in the late 1990s reported a discharge survival rate of 40%
(for victims with ventricular fibrillation treated with AEDs). Similarly, in a study
conducted within several casinos, trained security officers with AEDs readily available
garnered a survival rate of 74% for victims treated within three minutes of the cardiac
event. It is important to note that even at the level of greatly reduced response times (in
comparison to EMT response), in these domains, minutes were critical. In the casino
analysis, treatment after the 3 minute threshold produced a substantial decline in survival
rate to 49%.
�� ��������������
The AHA gathering of the 2nd Public Access Defibrillation (PAD) Conference
proposed four levels of defibrillation deployment resulting in a response taxonomy. To
align with the content of this dissertation, CPR and defibrillation are considered together
as part of the larger BLS function. One further addition, paramedics/EMTs are included
in the first level of response (they were not so designated by the conference). The merits
of each level are discussed in terms of quality and timeliness of response.
L1: Traditional first responder defibrillation: [paramedics, EMTs], police, or firefighters.
Level one represents the preeminent responders and can be expected to provide
the highest quality resuscitation performance. The problem is ensuring they arrive in
time to make a difference. Response times severely jeopardize survivability. Valenzuela
et al. (2005) report professional response times of 6 minutes 27 seconds from 911 call to
arrival at patient side with an additional 54 seconds to defibrillation. EMS services in
Dade County required almost eight minutes (7:56), likely a contributing factor in their
declining survival rate. Attempting to reduce the response tine , the county implemented
a program in which AED-equipped police units were dispatched simultaneously with
EMS. Response time improved to 4:53 (Myerburg, Fenster, Velez, Rosenberg, Lai,
SKurlansky, Newton, Knox, & Castellanos, 2002). In this study, the time of the first
vehicle to arrive (either police or EMS) provided the arrival data point. It would be
difficult to maintain such a redundancy for long periods to continue the reduced response
time, however.
L2: Nontraditional first responder defibrillation: life guards, security, flight attendants.
Members of these groups have generally achieved superior response times and
survival outcomes throughout the literature. Note the natural boundaries of the domains
and the ubiquitous authorities with mandated training. These contexts demonstrate the
potential for great success benefiting from the presence of capable responders already at
the scene of the SCA.
L3: Citizen CPR defibrillation: citizens with AED training
Programs to train citizens to respond to SCA events have been shown to increase
the number of victims that receive CPR, reduce the time to defibrillation and improve
survival (Hazinski, Idris, Kerber, Epstein, Atkins, Tang, & Lurie, 2005). A study in Italy
demonstrated an increase in survival rate from 3.3% (EMS) to 10.5% with trained (4 hour
course) volunteers (Capucci, Aschieri, Peipoli, Brady, Iconomu, & Arvedi, 2002).
Though this seems like a promising solution, it relies on citizens volunteering for
training. Willing volunteers do not make up a significant portion of the population.
Culley, Rea, Murray, Welles, Fahrenbruch, Olsufka, Eisenberg, & Copass, (2004)
reported that over a four year period, only 4,000 of the 1,700,000 people of King County,
WA received training (~0.2%).
L4: Minimally trained witness defibrillation. Individuals that witness SCA and have AED available but no training.
Perhaps this level was misnamed. It seems untrained witness would be more
accurate as the section fails to describe training of any kind. In these situations, the
response is rapid (associated with high quality outcomes), but its efficacy dubious. Here
is an opportunity for JITS! With an equipped and willing (though previously untrained)
responder in place, JITS can guide the operator through the procedure. The benefit of
device deployment no longer is impeded by the miniscule number of classroom-trained
volunteers. The instruction and course of action are tailored to the context. Thus, the
response approaches the level of L2, minimizing response time and elevating
performance.
��������������������� ��� ��On November 28, 2005, the International Liaison Committee on Resuscitation
(ILCOR) released 2005 International Consensus on Cardiopulmonary Resuscitation
(CPR) and Emergency Cardiovascular Care (ECC) Science With Treatment
Recommendations via the American Heart Association’s (AHA) journal Circulation.
With an emphasis on evidence-based medicine, they reviewed an exhaustive array of
studies employing human and animal models. The goal was to marshal the clinical
evidence in a fashion that could enhance the protocol and improve survival. Two points
from the introduction of this erudite, comprehensive work are pivotal in relation to the
research here.
• “The most important determinant of survival from sudden cardiac arrest is the presence of a trained rescuer who is ready, willing, able, and equipped to act”. (ILCOR, 2005. p. III-3 )
• “[O]ur greatest challenge remains the education of the lay rescuer”. (ILCOR, 2005. p. III-3 )
The report contains further implications for JITS. “[W]e must increase the
effectiveness and efficiency of instruction, improve skills retention, and reduce barriers to
action for both basic and advanced life support providers”, (ILCOR, 2005, p. PIII).
Many instances of SCA occur in public spaces (Becker, Eisenberg, Fahrenbruch
& Cobb, 1998), but with no trained responders present. Thus, these events occur with
people in the vicinity that could respond quickly, however they are ill prepared to act.
The contrapositive holds. Those capable of delivering appropriate actions require time to
arrive at the scene subjecting the victim to the descending probabilities of survival.
As a result, most SCA victims die outside the hospital without receiving
treatment (relating to Zipes & Wellens (1998) conjecture that 75% of victims die at
home). JITS systems in the home could drastically reduce the number of victims that left
untreated. A spouse, a child, a neighbor could be empowered to save a life with a JITS
device by providing life sustaining intervention while professional help is enroute.
In an effort to augment layperson resuscitation, ILCOR (2005) adjusted the
guidelines to be more effective and easier to learn. The guidelines increase the number of
compressions per minute to 30 from 15 and apply it to all victim types (except infants,
this reduces the need to memorize multiple ratios for different victim types). This change
is a small step in the right direction. While it is unclear that the number thirty is any
easier to retain than the number fifteen, the increased number of chest compressions
should increase circulatory function and maintain higher thoracic pressures over the
former protocol. But how much of a gain can be expected from these minor adjustments?
The conclusions of the ILCOR team clearly emphasize the need for immediate
response and improved technique. JITS designed systems provide a means to accomplish
both. JITS implementation could lead to dramatically reduced response times with
widespread deployment of a device that can instantly train the witness of a cardiac event.
Providing adequate ventilations and compressions in a timely fashion contributes to
healthier outcomes and improved survival rates. Sufficient deployment of support tools
could engender immediate, effective response through the ability to “instantly train”
bystanders. Thus, the researchers embarked on developing a CPR/AED system to test the
JITS framework. The following pages demonstrate the feasibility of garnering effective
treatment from untrained responders and provide an explanation of how JITS facilitates
performance.
BACKGROUND The advancement of technology in systems has elevated the demands placed on
the human operator. When system demands exceed operator resources, a problematic
mismatch emerges. The rapid development of new technologies and their deployment
across vast applications and domains makes keeping pace difficult. This discord is
exacerbated when a novice operator is called upon to perform the task.
The JITS framework strives to elevate user performance. Performance
enhancements are possible because JITS provides the user a workable plan, directive
cues, and corrective feedback. The system delivers the plan in accord with user progress.
The cues initiate user actions to begin each subtask. Those actions are closely
monitored, along with system state, to generate customized feedback and evaluate the
plan. All of this occurs while the user is engaged in the task.
To create such a system demands numerous coordinated efforts. The researchers
must establish a theoretical base capable of generating predictions, and robust enough to
account for a considerable range of human-system interactions. The technological
aspects must be developed, implemented and integrated with a systems engineering
perspective. Lastly, an environment must be fabricated in which to test theoretical and
applied aspects, including methodological constructs, data collection, and recruiting and
training participants.
Leveraging knowledge and experience from other scholarly domains is a prudent
start. Intelligent tutoring provided the interaction paradigm for JITS and has a rich
history of success. The research in expertise, feedback, and visual displays provided vital
details in how to deliver information to the user. The formidable task analysis literature
enabled the identification and assessment of task requirements and resources. Finally, the
Contextual Control Model (COCOM) offered a predictive model of human performance
within a JITS system as well as parameters for measuring outcomes. Intelligent Tutoring
Systems (ITS).
������������������������� �������������������������������� �������������������������������� �������������������������������� �����������The development of Intelligent Tutoring Systems (ITS) can be traced back several
decades. Early systems can be thought of as computerized flash cards with an ability to
score student responses. These programs generated problem sets (often in arithmetic or
vocabulary), recorded students’ answers and evaluated performance (Uhr, 1969).
The systems soon became more sophisticated. One of the first notable intelligent
tutoring systems was SCHOLAR (Carbonell, 1970). SCHOLAR tutored students in
South American geography. This was a landmark development in that it was the first
tutoring system that moved beyond a scripted hierarchy of knowledge and presentation.
Carbonell (1970) not only wanted to create a system that was able to handle all possible
answers a student might submit (a challenge to earlier systems), but also empower the
system to appropriately respond to questions asked by the student.
Another important step in the development of ITS was embodied in the work of
Collins (1977). Earlier that decade Craik & Lockhart (1972) demonstrated that deeper,
more elaborate processing improved learning and retention. In conjunction with this
work, Collins (1977) developed a Socratic method of exploration for his WHY system.
A Socratic system is structured such that a student works to “discover” knowledge. This
method empowers the student to build principles from experience and individual cases,
through self-directed exploration of the knowledge base.
������������������� ���������Many ITS projects share a basic architecture consisting of the same elementary
components. Perhaps the most basic as described by Burns and Capps (1988) are the
“Expert Model”, the “Student Model” and the “Tutor”. These modules were also
essential to the development of the JITS framework.
Expert Model The expert model is designed to encapsulate all available knowledge for a
particular domain. Domains in ITS have included arithmetic – BUGGY (Brown &
Burton, 1978), programming language - LISPIT (Anderson, 1988), and electronic
troubleshooting – SOPHIE (Brown, Burton, and deKleer, 1982). Amassing the expert
model requires the daunting task of harvesting and representing all collectible knowledge
for the particular domain. Anderson (1988) warns, “. . . a great deal of effort needs to be
expended to discover and codify the domain knowledge” (p.22). This information is
usually assembled by teams of experts in specific fields. The expert model commonly
serves as the standard to which the student’s actions and knowledge is compared.
Student Model The student (or user) model contains the system’s representation of the user’s
abilities and knowledge of the subject domain. A single student model is not sufficient
because user proficiency changes over time. Not only is there an overall trend of
learning and improved knowledge over the as the student progresses toward expert, but
there are perturbations of the user’s proficiency across and even within a training session.
A critical function of ITS is to properly diagnose the user and identify apposite
assistance. If students perform below their normal baselines, the system must respond
according. For the system to accurately gauge a student, it must go beyond a simple
comparison to the expert model. One shortcoming of current diagnostic functions is they
only evaluate response accuracy in terms of the knowledge base (Linn, 1990; Marshall,
1990). This often accomplishes little more than the ability to categorize a student. It
offers nothing to aid learning and performance (Linn, 1990).
Tutor Model Without a tutoring model (e.g., instructional module, or pedagogical agent) an
ITS would merely administer and score tests with no ability to impart knowledge or
understanding to the user. The instructional module interacts with the expert model and
the student model to formulate relevant information to the user.
The tutor model evaluates the student model in terms of the expert model and
decides what information to present, how to display the information, and when to do so.
The tutor model is the most visible module for the user. Through the interface, the
pedagogical agent serves as the teacher, consultant, assistant, or coach.
Customizing Information for the User The quality, quantity and timing of assistance determine the efficacy of the
system. Unnecessary instruction may be perceived as a nuisance and potentially hinder
performance. Users are unlikely to assimilate information that exceeds their
comprehension, arrives after significant delay, or conflicts with their mental models.
Accepting that not all users learn at the same pace nor benefit from the same level
of instruction, customization is crucial. Brusilovsky and Hoah-Der (2002) developed an
exemplar system basing information delivery on user needs. Their curriculum tutored
students in a web-based C programming application (WADEIn).
WADEIn demonstrated several advances for assessing user proficiency as well
providing user assistance. To help build the user model, WADEIn tracked three
parameters: the number of times the user had seen a particular visualization, the number
of times a user had performed an operation, and whether the student identified the order
of operation correctly. This data helped construct a rating for learner proficiency which
they assigned 1.0 – 5.0. This grade provided a basis for selecting specific assistance
protocols. For example, if a user was rated 1.0, the assistance provided all sub-steps of
the particular task (per the expert model) and showed the tutorial animation in slow
motion. A user rated at 5.0 received no sub-steps and no animation. Such acumen in
diagnosing user proficiency and matching information needs can benefit JITS
development by optimizing the human-system interaction.
%�����������&�������������������%�����������&�������������������%�����������&�������������������%�����������&�����������������������
A superficial comparison of intelligent tutoring and JITS may lead to the
fallacious conclusion of equivalence. Certainly, there are similarities – both are primarily
aimed at novices, are computer based, and leverage best practices of expert systems and
user-interface principles.
A major disparity arises however, simply examining the goals of each system.
The objective of ITS is to provide a student with knowledge and foster long-term
retention. In contrast, JITS systems are designed to support an operator complete a
specific task at hand. Thus, very different constraints and contexts apply to each system.
These differences are elucidated in terms of their objectives, time available, processing
strategies, and system input. A brief summary of these discords is displayed in Table 1.
�� ����� �
The disparate objectives of each system are yoked to their temporal constraints.
The extended time horizon of ITS enables the goals of learning, long-term retention and
fosters accuracy in the speed-accuracy trade-off. In contrast, JITS is not focused on
learning; retention is inconsequential. The objective is to perform the task to a
satisfactory degree in the limited time available. Due to the temporal constraints, it is
more important to sufficiently complete all requisite steps in the given time than to strive
for flawless performance on some tasks while failing to complete others.
Paradigm Objectives Time Available Processing Strategies System Input
Intelligent Tutoring Systems
• Long term retention • Learning • Accuracy over speed
• Long time horizon • Self paced • Low time pressure
• Deep/elaborative • Adaptive, transferable • Declarative
• Passive collection • Requires user input to computer
Just-in- Time Support
• Short term performance • Complete task • Speed over accuracy
• Short time horizon • Externally paced • Significant time pressure
• Shallow / perceptual • Mimicking • Procedural
• Active collection • Monitors user actions
����������������
There is generally an absence of time pressure while students engage in ITS
learning. The user determines the start time, duration, and sets the pace of the
interaction. In contrast, a JITS task will be subject to considerable time pressure due to
the criticality of the task (a non-critical task could tolerate awaiting the arrival of an
expert), making the task externally-paced. The JITS operator has a limited time envelope
in which to work compared to a self-paced, multi-session training exercise (the putative
ITS mode). Success or failure will be realized shortly in JITS facilitated tasks.
Table 1. Fundamental Differences between Just-in-Time Support (JITS) and Intelligent Tutoring Systems (ITS)
!��� ��� ������ � A significant discrepancy is also evident in the level of mental processing
indicative of each system. As mentioned above, learning has been shown to benefit from
a deeper level of processing (Craik & Lockhart, 1972). Intelligent tutoring systems
utilize declarative knowledge permitting extensive mental associations and abstractions.
This more elaborative processing strategy serves not only long-term retention, but also
pliant, adaptive application. Just-in-time support systems target a more perceptual level.
As a consequence, mimicking-type strategies, though inferior to elaborative processing
for memory, are efficient for accomplishing procedural tasks without practice.
�� ����������The final pivotal difference discussed concerns the inputs to each system.
Traditional ITS requires a direct manipulation by the user (type, point and click, etc.).
The system itself is passive and requires direct input by the user. An advancement
distinguishing JITS is its ability to actively collect data from the environment. With
various sensors the JITS system identifies changes of state and immediately updates its
models. This is transparent to the operator (sensors are integrated with the tools used),
and drives updated cues and feedback. Feedback and plan adaptability rely upon the
external data the system is able to collect and integrate. The operator is permitted to act in
a goal directed manner on the world (essential for performing a task), and relieved of the
additional burden of updating the system.
'�����������������'�����������������'�����������������'����������������������"�������#������������$����������$��
Hollnagel (1993) describes a continuum of human control that can serve JITS
development in both design and evaluation. The COCOM provides designers a tool to
identify parameters and determine control characteristics. This not only enables
prediction of control conditions, but establishes means to manipulate control states.
Additionally, the COCOM provides an effective assessment tool to evaluate performance
hypotheses.
Before describing the nominal control modes, it may be helpful to explain the
parameters that characterize a given state of control. The control modes will vary on the
following dimensions. Determination of outcome: the operator’s ability to detect and
interpret a change in system state. Subjectively available time: the time pressure
perceived by the operator. Number of simultaneous goals: the number of objectives the
user can maintain concurrently and their relevance to the overall goal. Availability of
plans: the user has access to heuristics, plans, procedures or something rule-like to guide
actions. Event horizon: is composed of the “history size”, which refers to the amount of
previous information utilized in a given decision, plus the “prediction length”, which is
an extrapolation of the future state of the system. Mode of execution: there are two basic
modes: a “subsumed” (ballistic or feedforward) mode where actions are executed
automatically, require assumptions and predictions, and a “feedback” mode in which
state data guide future actions.
Control Modes The control continuum is anchored by an absence of control at one end, and
highlights several milestones as control progresses to a very high-level, effective process.
Hollnagel has identified four characteristic regions of control to serve as watermarks on
the continuum. From least control to greatest control they are: scrambled, opportunistic,
tactical, and strategic.
Scrambled control resides on the low end of the continuum, and in its most
primeval, is akin to no control at all. The next action of the user is random and
unpredictable. Often, a blind trial and error approach is adopted and the determination of
outcome is extremely limited. User actions are often incongruous with the situation. A
user operating in a scrambled control mode has no heuristics, rules or procedures
applicable to the situation. The user is often under a great deal of stress due to workload,
time pressure, and a futile understanding of the system and/or the current environmental
conditions. The user is commonly in a panic state when operating in a scrambled control
mode.
Opportunistic control is the next region on the continuum featuring some ability
to guide the system toward the goal state. As in the scrambled control mode, the operator
perceives significant time pressure. There is still scant planning of the next action, but
actions are no longer characterized by randomness. Despite the lack of planning,
cognition is on the rise. This fosters the hallmark of opportunistic control: an ability to
recognize and act on salient cues in the environment. The user has moved from a
subsumed mode of execution to a feedback mode. Feedback is accessible and
interpretable. Additionally, if the environment is somewhat familiar, the user may be
influenced by frequently used schema.
Tactical control finally offers relief from the perceived time pressures and permits
short range planning because the event horizon is expanded both backward (previous
states) and forward (predicted states) in time. Performance is largely based on rules or
procedures available to the user. The operator can begin to anticipate needs of the near
future, but these predictions are constrained by the present context. The meanings of the
outcomes are also more completely understood (in terms of the current context). Two or
three goals may be active at once, and likely there is a plan, rule, or procedure to support
each goal. Feedback is again a valued input, but for a different reason. Feedback in the
tactical control mode is utilized for comparison with higher level goals.
Strategic control is a state of high stability and planning well beyond the
immediate context. Performance is usually effective and robust. The event horizon is
further extended (in both directions). The operator has the opportunity to contemplate the
highest level goals. Interestingly, both modes of execution are exhibited (due to the
long-range planning, there can be a significant lag in feedback and the need to take some
actions in the subsumed mode). Finally, it is also suggested that to be in a strategic mode
requires operator motivation. The user must embrace the additional cognitive load
required (for extended planning, reasoning, and observation) to attain a strategic mode of
control. As a consequence of the arduous workload, strategic control often cannot be
maintained for long periods.
Mode Transitions The movement between control modes is also of interest. Transition between
states can be discussed in terms of changes in the parameters outlined above. Since those
characteristics describe a state of control, it is appropriate to speak about an operator
moving along the control continuum as result of a change in one or more of those
parameters.
Parameters Scrambled Opportunistic Tactical Strategic Determination of Outcomes Obscured Limited Context
dependent Elaborate + prediction
Subjective Time Pressure Severe Significant –
Severe Light- Moderate None
Simultaneous Goals 0-1 1 2-3 2-many
Availability of Plans
None employed Minimal
Most goals supported by plans
Plans and contingencies available for all
Event horizon
Present only
Some history, little planning
Planning (based on current), use of previous data
Extensive planning & use of historic data
Mode of Execution Subsumed Feedback Feedback Feedback +
subsumed Action Selection
Random
Cue driven Plan driven Prediction
driven
The most pedestrian transition to envision is a simple step to an adjacent region.
This shift may be either an increase or reduction of control effectiveness. Examples
include tactical to strategic and opportunistic to scrambled. However, larger jumps are
also possible. Imagine an operator in a tactical control mode that suddenly encounters a
novel crisis. The subjectively available time may have all but vanished. The user may
have no heuristics to employ. The determination of outcomes proves elusive. All
planning has ceased and the only goal may be to figure out what just occurred. Clearly, a
scrambled condition has resulted. Similarly, it may be possible to jump straight from a
scrambled mode to a tactical mode. Though it may be considered extremely fortuitous, a
random action could bring the system to a stable, recognizable state in which the operator
has sufficient time, knowledge, and experience to assert tactical control. Hollnagel
(1993) does insist there is a constraint in moving to and from strategic control. Strategic
Table 2. Summary of COCOM parameters and control states (adapted from Hollnagel, 1993)
control can only be reached from the tactical region. Further, strategic control can only
be degraded to tactical (from which of course a fall to scrambled control is possible).
Knowing the characteristics of each control mode, it is possible to identify the
region in which an operator is working. Knowledge of the task, tools, and context
permits a designer to make mode control predictions. These parameters provide a
workspace that can be manipulated in an effort to improve an operator’s control. For
instance, a plan could be made accessible where a user had none. The absence of salient
features can be rectified to guide action. Feedback could be presented to promote
interpretation. These are just some of the steps that may be taken to improve
performance according to the COCOM. The priority of the parameters and combination
effectiveness will likely vary by task, user, and context, and are probably best derived
empirically.
Lastly, the COCOM, as it is a model of human performance, is a valuable tool for
not only design, but also assessment of a JITS system. It provides valuable
characteristics to describe the task and desired performance. These characteristics enable
researchers to predict performance in a given situation. Finally, the parameters can also
serve as dependent measures for comparing design decisions, methods and even full-scale
systems.
COCOM Implications for JITS design
• Provide a plan of action, adaptable to dynamic conditions and needs
• Provide salient, action-directing cues
• Provided interpretable feedback to correct actions and update system state
• Work within time constraints
• Control mode is a function of parameters above; designers have power to manipulate
�#��� ��In order to assuage potential ambiguity, a brief definition of expertise is offered.
An expert is a person that has gained extensive knowledge and prodigious skill in a
particular domain. The knowledge is also highly organized and accessible by the expert.
Anderson (1983) insists that both theoretical and experiential knowledge are required to
become an expert. Ample time is another correlate as a common finding suggests it
takes ten years to become in an expert in many domains (Klein, 1998). For a treatise on
expertise, see Sternberg (1997).
Expertise is a major component and is ubiquitous throughout JITS. First, task
knowledge must be collected in order to prescribe a successful method for task
completion (the plan). Then, domain expertise is required to create effective, adaptive
protocols capable of supporting numerous scenarios. To produce a robust system, the
experts must anticipate failures and complications caused by the system, the user, or the
environment. Ultimately, this vast, high-level knowledge must be translated to a form
suitable for the non-experts supported by the system. Thus, it is critical to examine the
documented differences between experts and novices.
The literature has largely focused on what expertise is. By default a person
lacking these superior skills and knowledge is considered a non-expert. This paper will
not demarcate the categories and qualifications of non-experts in great length utilize a
common ITS-employed continuum: novice, beginner, intermediate, expert (Virvou, &
Moundrido, 2001). The current state of JITS is aimed at the less skilled of this set, but it
is believed that JITS can be customized to service more advanced operators as well.
Characteristics of Experts A substantial amount of research is available characterizing experts and expert
performance (Chi, Glaser, & Farr, 1988,). Glaser and Chi (1988) have established seven
general attributes representative of experts presented in table 3.
Expert-novice differences With the focus of the literature directed at experts, differentiation between experts
and non-experts is often left to inference. Fortunately, these conclusions are warranted.
Johnson (1988), in multiple domains, has observed that novices do not have extensive
domain knowledge, and cannot encode information well (e.g. they have representation
and pattern matching difficulties) nor process new information quickly. Miller & Perlis
(1997) also discussed novices’ inferior knowledge base and structure, adherence to
superficial cues, and utilization of small, fragmented information units. Not only did
novices utilize more superficial knowledge, but the knowledge base lacked cross
referencing and the organization experts are able to impose.
Therefore, in developing a system for non-experts, the inverse of each axiom
above can be used to guide design. Different aspects of the JITS framework were crafted
1. Experts excel in their own domain.
2. Experts perceive large meaningful patterns.
3. Experts solve problems quickly with little error.
4. Experts have superior short- & long-term memory in their domains.
5. Experts represent problems more abstractly.
6. Experts are able to spend great deal of time analyzing a problem qualitatively.
7. Experts have strong self-monitoring skills.
Table 3. Expert characteristics (Glaser & Chi, 1988)
to specifically address these salient needs. Below are the design (D) recommendations
pertinent to each expert characteristic described above.
Novices:
are outside their domain (or are experiencing a novel/low frequency event) D1. Novices do not excel in the given domain, they need help. That’s the
motivation for JITS!
cannot decipher meaning in complex patterns D2. Information must be parsed into simpler, more digestible chunks. Critical
parameters and combinations must be made salient. The system can help the user construct patterns in small increments.
are slower and more prone to error
D3. Along with reduced data size, the pace at which information is delivered should be slowed for novices. The sequence may also need altering. Repetitions may be necessary. The pedagogical model should monitor and control the pace and sequence pursuant to operator needs.
have no strategies/knowledge to enhance STM & LTM
D4. The system should relieve burdens to STM & LTM by holding the information and making it visible, accessible, and congruent with the current subtask.
don’t understand abstract concepts of the domain
D5. Cues and feedback must utilize concrete representations. Designers can’t assume knowledge on the part of the user. Metaphors, higher level concepts, and domain knowledge are likely to be lost on a novice.
won’t analyze the problem
D6. Novices won’t have time or ability to analyze the situation on a deeper level. The context-aware system will be responsible for identifying the proper course of action given the context.
will not be able to self monitor reliably if at all.
D7. Novices will likely be far too stressed to have any resources available for this meta-task. Smart algorithms and sensors take over this job. In conjunction with the situation assessment (D6), the system monitors operator performance in order to optimize cues and feedback to assist the user.
The emphasis in employing these principles is that it is imperative to provide
information congruent with the operator’s knowledge. It is crucial to design systems
suited for the user’s’ level of expertise and these guidelines are intended to illuminate the
specific needs of novice users. In a human-to-human tutoring example, findings by
Hinds, Patterson and Pfeffer (2001) demonstrated that experts used more abstract (and
fewer concrete) statements as they tutored novices. In contrast, novices tutored by
beginners performed better and reported fewer problems with the instructions. This was
largely the result of beginner tutors using more concrete terms, and their ability to
comprehend the novices’ states of learning.
Developers of JITS systems should not be obfuscated with the seduction of
attempting to transform novice users into experts. Tasks may consist of only a single
trial, not nearly enough time to establish expertise. By abandoning the effort to create an
expert and embracing the task of mimicking one, several advantages arise. A shallow
level of processing can be targeted. While inferior for retention, shallow processing
reduces the demand on the novice’s limited cognitive resources and leverages more
primitive processes such as perception-action mechanisms and preattentive processing.
As Klein (1998) elegantly suggests, experts see the invisible. The crux of JITS
development is to unveil and communicate that information for novice use.
Expertise Implications for JITS design
• Break information down into comprehensible chunks
• Use concrete (as opposed to abstract) representations
• Ensure the pace of information delivery is apt for user population
• Minimize cognitive burdens (memory, search, attention capture) as much as possible.
• System should assume monitoring role
• Keep information visible
%������&�Feedback, loosely defined, may be considered any information provided back to
the operator concerning performance or outcome. This discussion will exclude intrinsic
feedback (that which comes from within the operator), but focus on the information
provided by the system as a result of operator actions. Many terms such as knowledge of
results (KR), knowledge of performance (KP), augmented feedback, and intrinsic
feedback have been used throughout the literature to describe different effects of
feedback (Proctor & Dutta, 1995). In the following section, specific terms have been
preserved where research findings are reported. Their nuances are less important for the
larger discussion as all contribute to the design of effective feedback for JITS.
In order to provide closed-loop control, feedback must be communicated to the
operator. As discussed in the COCOM section above, feedback enhances operator
control. Of course, the feedback presented must be interpretable and actionable for the
operator. Generating suitable feedback for non-experts often requires a skillful balancing
of contradicting design issues.
Benefits Reviews and meta-analyses have demonstrated that accurate feedback generally
improved overall performance in a variety of tasks (Azevedo & Bernard, 1995, Kluger &
DeNisi, 1996, Salmoni, Schmidt, & Walker,1984). Reliable Feedback has been shown
to enhance learning and performance compared to no feedback or misleading feedback
conditions.
Concomitantly, erroneous feedback has been shown to produce deleterious effects
(Young & Lee, 2001). These findings demonstrate stout support for the use of reliable
feedback in developing JITS systems. In accordance with the JITS tasks, Azevedo
(1995) has emphasized the importance of feedback in a computer supported environment.
In support of immediate task facilitation, Goodman (1998) concluded that
frequent and immediate feedback to participants improved their performance during
practice but did not report the same benefits for retention. Here, practice refers to the
participants’ first encounter(s) with the task. Similarly, Young & Lee (2001) found that
more feedback in the “acquisition phase” facilitated task performance (but not retention).
In JITS supported tasks, “practice” occurs simultaneously with the “test”. Therefore,
frequent and immediate feedback will likely generate desired effects for JITS systems.
An additional benefit, feedback has been found to increase participant motivation.
Participants often work harder, show more interest in the task, and persist longer than
those not receiving feedback. Interestingly, Salmoni, Schmidt, & Walker (1984) found
these effects persisted for some time even after removing KR (e.g., feedback).
Pitfalls Not all research has witnessed positive returns from feedback. Some studies
resulted in null or negative effects, and other studies showed greater learning with
reduced feedback. Negative effects can often be traced to poorly designed feedback as it
relates to the task or the operator. For example, feedback may not be suited for the
operator’s level of expertise. “Cognitive feedback” attempts to provide individuals with
considerable insight to allow them to learn relationships between the cues they utilized
and the judgments that followed. This however, required significant domain knowledge
and was not suitable for novices (Balzer, Hammer, Sumner, Birchenough, Parham, &
Raymark, 1994).
In complex tasks, multiple sources of feedback may provide contradictory
information and degrade performance (Goodman, 1998). Thus the reliability of the
system must be sufficient to supersede conflicting data. Efforts should be made to reduce
the overall data stream in an attempt to provide only what is necessary to facilitate the
task.
Other problems relate to the timing of the feedback. Rapid feedback may elude
operator detection. Similarly, the operator could be engaged (mentally or physically)
with a task and unprepared to receive new information. Feedback presented at such times
will not be processed or incorporated into the next actions. Further, if concentrating on
the action itself, the user could view the feedback as interruptive and distracting.
However, feedback lag may also present a problem as Salmoni, Schmidt, and Walker
(1984) showed that a long delay in feedback (as opposed to KR after each trial) degraded
practice performance (but facilitated long-term retention).
Other researcher (Young & Lee, 2001) also showed benefits of withholding
feedback. They used a “bandwidth” feedback condition in which feedback was only
communicated if performance deviated from a specified tolerance (10% error). They
found this group to outperform the group receiving feedback after every trial. In the
same study, they found that over specified feedback (e.g., unnecessary precision), was
usually ignored by the participants.
Feedback implications for JITS design
• Customize feedback pursuant to operator knowledge • Feedback provided must be reliable
� Conflicting information can confuse the user
� Excess precision (based on user knowledge) will likely be ignored
• Feedback timing is critical
� User must be able to relate feedback to relevant activity
� User must be ready to attend to feedback information
• Feedback based on threshold may be more suitable than after every trial
• Feedback may serve as motivation in completing the task
' ����( ���� ��As most of the tasks supported by JITS will entail significant complexity, the
visual channel could be overloaded with information. Technology-intense domains
typically require visual displays to support detection and interpretation tasks, particularly
when governed by capricious constraints (Sanderson, Haskell & Flach, 1992). This holds
for the CPR/defibrillation task investigated here.
Park and Hopkins (1992) outlined six conditions in which dynamic visual
displays (DVDs) are effective. The majority of those stipulations are highly cogent with
JITS system development to support CPR. The use of DVDs is effective when:
- demonstrating sequential actions in a procedural task : The sequencing of the
procedural steps is achieved through context-dependent algorithms based on the AHA
protocol.
- obtaining attention focused on specific tasks or presentation displays: The system
choregraphs the responder’s attentional demands, switching between information, tools,
and the victim as required.
- illustrating a task which is difficult to describe verbally: It would be quite difficult
to verbally impart hand configuration and placement in performing chest compressions.
- explicitly representing invisible system functions or behaviors: A responder would
have no way of knowing their inspired volume. The system tells them to give more
breaths if insufficient.
Animation Animated graphics (e.g., dynamic visual displays) are a great tool for conveying
complex, temporally constrained, spatial information. It is envisioned many JITS tasks
will require space-dependent manipulations of artifacts in the workspace. For users
unfamiliar with compulsory components, spatial relationships, or procedures, animation
provides strong cues for comprehension. Reiber (1990) asserted animation provides three
vital attributes for performance: visualization, motion, and trajectory. ChanLin (2000)
demonstrated that spatial and procedural elements of animation play an important role in
interpreting information
Before trumpeting the success of animation, it must be noted that the animation
literature is tainted with mixed results ( Kehoe, Stasko, & Taylor, 2001, Park & Hopkins,
1992, Rieber, 1990, and Tversky & Morrison, 2002 for reviews). One problem with the
research is the difficulty constructing fair comparisons (Tverksky & Morrison, 2002).
They further observed that, “animations are often too complex or too fast to be accurately
perceived” (p. 247). They suggested much of the animated designs violated the
“apprehension principle” requiring graphics to be conceived appropriately and perceived
accurately.
An example of this excessive complexity hypothesis was captured in an
experiment by ChanLin (2001). A procedural learning task in physics (resultant force
vector) was taught to high school students using text, still graphics and animation. The
results indicated that still graphics were significantly better for the novice group (over
text or animation), while animation garnered superior performance from experienced
students. They concluded the animation was too complex to support novice
comprehension and required substantial prior knowledge to be interpreted correctly.
A similar phenomenon was observed when graphics-to-text comparisons were
being made in the 1970s. Early studies failed to uncover the benefits of graphics over
text. Willows (1978) argued that performance suffered when graphics were superfluous,
complex, or incongruent with the text. Eventually, greater effort was expended on the
design and graphics began to improve performance. By mid-decade, Booher (1975), had
sufficient data sighting the superior performance of well constructed graphics-plus-text
presentations over either alone. Today, it is accepted that well contrived graphics are
highly effective at portraying visuo-spatial information (Tversky, 1995).
A number of experiments describe the benefits of animation for elevating first-
time performance, vital for the single-trial nature of JITS tasks. One such experiment
comparing textual versus animated instructions provides a convincing illustration.
Palmiter and Elkerton (1991) examined user ability to learn a procedural task on a
desktop computer. The procedure involved a training period, and two tests of retention:
The retention tests favored the group receiving textual instructions. It was concluded that
deeper encoding required for reading and processing textual propositions benefited users
on retention tests manifest in the text group’s dominance in the delayed post-tests.
However, in the training trials, the animation group demonstrated a superior
ability to achieve early success. The animation group performed their training session
tasks in approximately half the time the text-based group required. Intriguingly, this was
not at the expense of accuracy. In training, the animation group surpassed the text group
by performing over 90% of their trials correctly, while the text-based group failed to
achieve 80%.
A conclusion drawn by the researchers (and supported by comments from
participants) was the animation group adopted a mimicking strategy. They simply
followed the animated procedure, mimicking tasks and actions with little mental
processing.
In JITS systems, the training and test occur simultaneously. The CPR task
required immediate action by unpracticed individuals in a critical moment. There is no
need for retention or even comprehension, simply the aptitude to follow the procedure.
Animation can be beneficial in conveying task information for unpracticed, time-
pressured tasks.
Visualization (DVDs) implications for JITS design
• Important for conveying dynamic visuo-spatial information (difficult to verbalize)
• Appropriate for sequencing and procedural information
• Effectively portray motion and trajectory
• Animations should be simple and relevant
• Pacing of dynamic visuals must be carefully considered
�� &������ ������A critical step in developing JITS systems is a thorough examination and
description of the tasks, procedures, and goals. Task analysis (TA) yields a deeper
understanding of the fundamental elements of the task and exposes the nature and
organization of the sub-tasks. Through task decomposition, requirements assessment,
and error prediction, task analysis elucidates critical performance parameters engendering
efficient, fault-tolerant systems.
In general, a “Task Analysis”, serves to marshal the task-relevant design
implications. A surfeit of specific task analysis techniques exists including: cognitive
task analysis (CTA), hierarchical task analysis (HTA), critical path analysis (CPA),
timeline analysis, failure modes and effects analysis (FMEA), and goals-means task
analysis (GMTA). This section does not detail the various methods available or delve
into the current TA issues (see Annett & Stanton (2000)), but instead serves to call
attention to their importance and breadth.
Selection of specific techniques should be driven pursuant to the focus of the
analysis and researchers should leverage methods for their strengths and suitability in the
context of the project. The research plan may require a focus on: physical actions,
cognitive requirements, performance evaluation, temporal/sequencing issues, functional
descriptions, or goal accomplishment. Kirwan and Ainsworth (1992) provide a
comprehensive summary of the strengths of more than two dozen task analysis
techniques.
Task Decomposition Task decomposition is a widely employed technique often entangled with other
tools such as a Hierarchical Task Analysis (HTA) or Goals, Operators, Methods, and
Selection Rules (GOMS). Task decomposition requires the identification of the
constituent activities that, when performed together (and correctly), result in the
achievement of a higher goal. Effective task decomposition yields a clear description and
understanding of all pertinent sub-task activities. Through task dissection, the analyst can
gain the greatest insight to task requirements and constraints. Task parameters
commonly exposed by decomposition include: atomistic action descriptions, physical
and cognitive demands, failure points, temporal constraints, commencement and halting
cues, decision thresholds, resources and constraints, and performance criteria.
Hollnagel (1993) disambiguates a few terms to facilitate TA discussion. A goal is
a state that is reached when tasks have been successfully completed. A Task is a
collection of actions that are used to achieve a goal. Lower-level actions are known as
task steps, which begin to capture fundamental actions. A pre-condition is a requirement
that must be met in order to attempt a task or a task step. Not surprisingly, a goal can
have multiple tasks and sub-tasks with several pre-condition requirements. Further, the
absence of a pre-condition may result in the need to create a goal satisfying a pre-
condition which could then require tasks and sub-tasks to realize that pre-condition. This
can rapidly burgeon into an intricate, nested hierarchy. There are also cases in which
tasks are not dependent upon subordinate tasks (e.g., “pure” tasks), and are sequence
independent. That would represent a non-hierarchical task description.
What constitutes a pure task will largely depend on the level of analysis. This
granularity decision resides with the analyst. What precision must be reached by the
analysis? This can be a difficult question to answer and should be determined in the
context of the project. A guideline that may be helpful is to assess what can be reaped
from the requirements assessment and error prediction exercises. If the results of these
seem satisfactory, then the level is likely sufficient. If there are obvious voids, the level
of analysis may be too shallow. Conversely, is there anything to be gained by excessive
analysis (i.e. nerve innervations)? Again, these questions must be addressed in the
context of a specific project.
Error Analysis Though task decomposition can be an arduous process, the benefits are quite
valuable. The results can be described in terms of a “needs analysis”. These needs will
be crucial in the prediction and analysis of erroneous behavior. Error is a relative term,
and is generally determined in comparison to an expected “correct” performance (based
on goals and known successful activities relevant to the goal). Several of the task analysis
tools available are apposite for error and consequence prediction. A few examples are
barrier analysis, FMEA, fault tree analysis, GOMS, and Petri-nets (Dix, Finlay, Abowd,
& Beale, 1998). Again, the researchers should employ particular techniques that
complement underpinnings of the research.
A prudent methodology for detecting error is searching for inconsistencies
between the needs required to perform the task and the capacity to meet those needs. The
capacity is composed of operator capabilities/performance, resources available to the
operator, operator-system interaction, and system/environmental constraints.
Deficiencies in any of these areas can impede progress toward the goal state. Each of
these areas is subject to considerable complexity which is proportional to the level of
detail. Further, the integration of these components to aggregate a system-wide
assessment is nontrivial. However, this effort to predict and mitigate errors in the design
stage may help evade significant consequences. Of course, if a system already exists (or
an analogous system), much can be gleaned through observations.
Task Analysis implications for JITS design:
• Break down tasks into elementary actions that build larger tasks and actions
• Assess the cognitive and physical resources needed to complete each subtask
• Assess the means to meet the demands
� Determine resources supplied by the system
� Determine resources supplied by operator
• (requires realistic evaluation of user population)
������������������������������������������������������������������ �������������� ��� �������������� ��� �������������� ��� �������������� �����
The cardinal elements of a JITS system are derived from user needs with respect
to task completion. The novice lacks domain expertise and therefore cannot formulate
plans, recognize decision and action cues, and will have little success monitoring the
progress of the task. The framework of JITS addresses these problems by developing and
communicating plans, cues, and feedback to the operator. Figure 1 depicts a generic
schematic of a JITS system. The diagram includes the JITS system, the operator and the
task space. Notice that no effectors extend from the operator to the JITS device. Instead,
the device collects input from the state space via tools (which may be manipulated by the
user).
Just-in-Time Support System
COCOM
Error Analysis
PLAN
Task Workspace
Human Operator
Task Analysis
Expertise
ITS
Visualization
FB
receptors
effectors
sensors
Algorithms
receptors
tools/equip
Modules
+
Sequencer
CUES
Figure 1. Schematic of a Just-in-Time Support System.
!��� �The first requirement of a JITS system is to generate an effective plan for the
operator. Lacking a starting point, a user may fail to begin the task or may perform
actions that are fruitless or counterproductive. The task and circumstances may be
overwhelming, disabling cognition in general and specifically executive function.
Providing the user a plan and assertive direction will assuage substantial cognitive
burdens.
Plan development is much more than generating a single recipe for action.
Sensors continuously monitor the workspace. Algorithms constantly assess the
progression toward the goal state. Decision rules are employed by the system to alter the
task as necessary. These processes eclipse the capacity of a novice and must reside in the
support system. Non-experts possess scant ability to recognize the need for
contingencies, and have few resources to derive them. Thus the plan is not just a ballistic
delivery of a single formulation of actions, but an adaptable confluence of actionable data
that facilitates task completion in the given context.
The adaptability of the plan greatly depends on the effort expended in the task
analysis. Sufficient specificity of the TA should produce elementary action modules.
These modules are the building blocks of information delivery. Basic action scripts are
presented to create almost any task-relevant action sequence needed. The modularity of
the task actions fosters flexibility as individual actions can be called at anytime creating
extensive sequencing options.
A meticulous task analysis remunerates JITS developers in another way. The
effort reveals the information needs to support the task. This obviously has a significant
impact on the design and inclusion of various sensors and algorithms assigned the tasks
of collecting and interpreting data (which impacts the selection of plans, cues, and
feedback). It is not difficult to imagine a scenario in which a system is deployed (based
on a substandard TA), and found to have informational deficiencies when tested with
humans. Anticipating information needs early in the process prevents unwelcome
surprises and design modifications later. Plan progression and alterations, as well as the
operator’s ability to follow them, are reliant upon discovering and satisfying the
information needs.
The optimal plan, if followed appropriately, will likely be at least somewhat
accordant with an expert’s sequence of actions. However, the measure of success is not
how closely the operator tracked the expert model, but an assessment of the actual end
state comapred to the desired end state.
There may be numerous paths to meet task objectives. The development of plans
requires significant hypothesizing and supposition by the designers. Since it would be
impossible to imagine all possible human actions, performance criteria must be
established to provide tolerances for assessment. Identifying acceptable ranges can serve
as decision points in the algorithms. For example, how many times should the system
repeat a cue before trying a new presentation? When is it time to abandon a subtask and
attempt another action? Of course these depend on the task, but tolerances need to be
identified and programmed to permit such decisions.
��� �In order to initiate proper actions, the system must engage the user prior to
providing guidance. As the COCOM model indicates, operators in the opportunistic
mode can utilize cues to improve their performance. A primary objective of JITS is to
provide those cues.
Presenting information without the operator allocating appropriate resources will
accomplish little. In many subtasks, the first cue should be audible. Omnidirectional
aural information may be necessary as many targets vie for the operator’s visual
resources. An aural indication can redirect the operator’s attention to the display. A
synthetic voice or recurring chime could apprise the operator of new information.
For some task elements, the auditory channel will be sufficient. For example, a
metronome-like aural cue can guide pacing without taxing visual resources. However,
many tasks employing JITS will likely entail significant complexity thereby requiring
visual cues for adequate support. Providing a consistent visual mapping of the
components on the display to their real-world counterpart will enhance object
recognition. Artifact identification, relative positions and motions, and landmarks, can
all be conveyed through visual cues. Operators can leverage cues such as color, patterns,
shape, movement, and spatial relationships to identify and manipulate objects correctly.
Cue development relies on adequate task decomposition. A simplified subtask
can reduce clutter compared with more complex action sequences. This limits the
number of cues presented simultaneously, relieving the burdens of search and
discrimination (extra cognitive tasks which can lead to error).
%������&� Section 4.3 reviewed the academic findings of feedback pertinent to JITS. This
section describes feedback as a pillar of JITS (along with plans and cues) and discusses
the functionality of feedback as a tool.
Feedback in JITS applications is analogous to a servomechanism. Optimally,
small corrections through effectively communicated feedback will direct operator
performance toward the goal state. Slight, qualitative adjustments can usher the user
along the desired performance trajectory. Pragmatically, cases will exist that require
significant deviations to current actions. This could include presenting alternative
formats of the same information, selecting a different sequence of subtasks, or activating
a completely new plan.
Feedback is also capable of providing additional means of imparting information
if the first cues were insufficient. The system may simply repeat the cue, or if
information is available, a new cue may be delivered to address a specific shortcoming.
The system may substitute alternate cues hoping to find a better match for the operator.
Ultimately, if a user is unable to adequately act on a cue, the system may abandon that
subtask and devise a new set of actions. Many of the guidelines outlined for cues are also
applicable to feedback design. Expertise, visualization, and of course the feedback
literature, are all prudently employed in the development of feedback presentation.
Examples include the need for concrete representations, apposite timing and sequencing
of information, and suitable decomposition.
Two objectives can serve to guide feedback design. First, feedback is necessary
to convey information for corrective actions. Regardless of the degree of error, the
feedback module carries the responsibility of error reduction. Secondly, updates of task
and system status should be communicated to the operator. Novice users will be ill
equipped to evaluate and track task success; their errors need to be corrected, but they
also need to be told when they get things right. Feedback provides the user with
important information to incorporate in future actions as well as serves as motivation to
adhere to the task.
(���������(���������(���������(������������� The researchers endeavored to promote design methods enabling satisfactory
performance by novice operators in unfamiliar tasks. The fundamental provisions of an
action plan, cues, and feedback can propel a naïve operator from a state of bewilderment
to a mode of control enabling information processing and action. However, the
framework required validation.
Non-expert responders performing CPR and defibrillation provided a robust
context in which to thoroughly vet the assertions of JITS. First it enabled the researchers
to demonstrate the dismal performance of those without training or JITS, establishing a
baseline. Second, JITS induced effects were examined on trained and untrained
responders. Further, the context afforded the utilization of a respected protocol (AHA),
strict control of training, interaction with technology, and quantitative and COCOM
based performance measures, all critical parameters of JITS. Thus the experimental
investigation was constructed around the Basic Life Saving (BLS) task of CPR and
defibrillation.
Below, hypotheses are presented by participant grouping. There are two distinct
hypothesis tracks. The first, labeled [CPR], captures predictions concerning participants’
physical performance related to the execution of CPR tasks as prescribed by the AHA
protocol. These tasks include delivering breaths and chest compressions and consider
variables such as the volume of breath delivered and the frequency of chest
compressions.
The second hypothesis track is devoted to classifying participant behaviors
[BEH]. COCOM based predictions foster the categorization of each participant’s
behavior as scrambled, opportunistic or tactical. In contrast to the physical variables
[CPR], that simply capture the result of their actions, COCOM measures [BEH] provide
insight as to how responders performed those actions.
GRE trained, no support (GRE-NO):
[CPR1] Performance measures will be grossly inadequate and far below the other groups.
[BEH1] Responses will be haphazard; they’ll have little understanding of their performance and make no improvements over the course of the scenario. They will demonstrate scrambled control.
GRE trained, device supported (GRE-DEV):
[CPR2] Early performance will be substandard but will ramp up quickly and should surpass the trained/unsupported group.
[BEH2] Most actions will be guided by the system. They will rely heavily on cues and feedback. May show signs of scrambled control early but plans, cues and feedback will quickly advance them to an opportunistic mode.
CPR trained, no support (CPR-NO)
[CPR3] Will perform well, but may make some mistakes. Thus, their overall performance could suffer somewhat, but expected to be adequate.
[BEH3] They must rely on memory. They have no cues or feedback. A plan should be evident if they have retained their training. This represents tactical control.
CPR trained and Device trained, both device supported (CPR-DEV) & (DEV-DEV)
[CPR4] These two groups should outperform the others groups. Overall performance should closely track prescribed treatment per AHA.
[BEH4] Though trained, it is likely they will rely or at least refer to the system commands and feedback, showing evidence of leveraging their training and the system. An opportunistic control mode is expected here.
METHODS The investigation employed a randomized 3(training) x 2(device) nested factorial
design. Naïve participants received one of three training programs and returned 2 weeks
later to complete the experimental procedure (either with or without device assistance).
Sensors collected multiple measures providing means to analyze participant performance.
#�����������#�����������#�����������#���������������Measures from 100 volunteers were analyzed for the study. The participants were
largely members of the University of Utah community. The mean age for all participants
was 22.6 (SD=6.7) years. We required that participants were not knowledgeable in CPR
or first aid. Forty-five participants claimed prior CPR training with a mean of 6.5 years
(SD=6.5) since the training (it is common that high school students from the area receive
a training class however, it does not result in certification). A questionnaire was
administered in order to screen for CPR knowledge (Appendix D). The researchers asked
if they had been trained in CPR or with an AED and asked basic skill questions such as
the ABCs of CPR. None of the participants was able to answer all of the knowledge
questions correctly.
Participation in this experiment required a tyro’s knowledge of CPR for two
reasons. First, two conditions sought to simulate highly uninformed responders. The
remaining three conditions were best served with controlled, equivalent training for each
group. Participants demonstrating intermediate CPR knowledge or greater or currently
certified responders were excluded. Four volunteers were excluded prior to training due
to advanced CPR knowledge. Two additional participants were excused after
experimental training because they became certified after during the delay interval (in
order to be employed as lifeguards).
������������������������������������!��������������
Three different training conditions were utilized in the experiment. The “CPR”
training consisted of a slightly modified version of the American Heart Association
(AHA) protocol. The modifications included dropping portions of the procedure
including checking for pulse and calling for help as these were not emphasized in this
investigation. Two groups of twenty (40) participants received CPR training. Twenty
(20) more participants received “device” training (DEV-DEV in figure 1). Device
training mirrored CPR training as much as possible but incorporated the device during
instruction. DEV-DEV was the only group with exposure to the device prior to the
experimental session. A registered nurse, (responsible for anesthesia resident training at
the hospital and currently certified as a CPR instructor), administered the CPR and device
training. Prior to completing the training, each individual demonstrated aptitude to an
adequate level and trained to criteria. Finally, two more groups (20 per group) trained in
a fashion unrelated to the task. The experimental design called for two naïve groups in
which no relevant training was given. These participants learned strategies to improve
verbal scores on the GRE in order to maintain a consistent experience for all groups.
Comparable class size, duration, and participation were maintained for all training
sessions. Figure 1 provides a visual representation of the design and names the groups.
TESTING Figure 1. 3 (Training) x 2 (Device) nested factorial design
�����������The research assistants providing analysis also required special training in order
to properly code the data from the video tapes. Coders were taught to distinguish
behaviors for given actions but kept uninformed as to the foundation for those
distinctions and remained naïve to the hypotheses, control mode categories, and
participant training.
A coding sheet (Appendix B) and specific coder training governed the
systematized coding method. The sheet consisted of an array of actions reflecting
potential responder behaviors for each fundamental subtask and coders were trained to
distinguish subtle discrepancies in execution. For example, the coders chose from three
descriptions of mask holding to capture performance data. One selection was an
anesthesia hold. This type of mask placement was taught in training and required a
TR
AIN
ING
GRE trained
CPR trained
n = 20 for each group N=100
no device device
GRE-NO
CPR-NO
GRE-DEV
CPR-DEV
DEV-DEV
specific hand and finger configuration easily recognized on video. A second choice was
any two-handed grip (less the intricacies of the anesthesia hold) that involved two points
of contact sealing the mask. Lastly, a catch-all was used to code inferior mask holds
such as one-handed holds or the failure to use the mask at all.
The coders compiled these raw behavior data enabling the creation of a table to
categorize each participant as behaving in one of three control modes (per COCOM,
strategic was eliminated). Once the coders assessed individual actions such as mask
placement, the researchers calculated the proportion of actions by mode for each
participant and a control mode category was assigned. Most participants performed
actions from at least two of the categories, but usually a dominant control strategy
emerged. Over the duration of the scenario, each judgment contributed to a control mode
profile fostering an aggregate classification of mode for each responder.
)�������)�������)�������)�����������The device provided a means for novice users to perform effective CPR and
defibrillation tasks. Based on the AHA protocol, it delivered protocol instructions crafted
for novice understanding via visual and aural prompts. Utilizing smart sensors and
algorithms, the device assessed user actions and customized feedback to improve
performance. For example, if the sensors detected the chest compressions were too
shallow and too slow, it prompted the user to “push harder” and “push faster” for the next
set of compressions. These sensors not only drove feedback algorithms but also collected
data. In addition to relieving many cognitive burdens, there were also engineered
improvements such as the integrated headrest and mask. Pre-experiment investigation
revealed many novice responders had difficulty maintaining an open airway while giving
breaths. The headrest was designed to ensure victim head-tilt to keep the airway open
without requiring the responder to manually perform the task each time while giving
breaths. Figure 2 shows the device in use.
Figure 2. The device in use on training mannequin
The system provided the first visual and auditory cues to initiate the actions of the
novice responder. The system governed the pace and content in accord with the operator
performance by monitoring changes in the task space (i.e. tool placement, flow meter
readings). For example, the instructions required for rescue breaths were withheld until
the system recognized correct head placement. Pressure sensor in the headrest
determined the placement of the victim’s head. The system managed the action plan and
provided the user simple, actionable commands.
After giving simple instructions and monitoring performance, the system then
provided feedback if necessary. For example, after placing the mask, the system stated
‘give two breaths” and provided an animation demonstrating the proper method. Then
the sensors actively monitored the inspired volume of air into the lungs. If the rescuer
delivered insufficient volumes, instead of moving to the next step, the system encouraged
the responder to “give two large breaths”. Feedback of this ilk was critical in elevating
performance early in the scenario.
A headrest, anesthesia breathing mask (with one-way valve), and defibrillator
pads from Zoll’s AED-plus, and Lilliput 8-inch touch screen LCD served as the tools
available to the responders in the “device” condition (Figure 3.). The headrest was
customized from the foam of Giro bike helmet. Two pressure sensors were inserted at
the neck and in the center of the bowl (in order to detect proper head placement). The
mask apparatus consisted of a standard anesthesia mask, bacteria filter and a one-way
valve (directs victim’s exhalation away from the responder).
A Dell Dimension desktop and a Dell Latitude laptop provided the computing
resources for generating the displayed animations and auditory cues as well as collecting
data from: two Novametrix Medical Systems, Inc. CO2SMO Plus flow monitors (one
inside the mannequin, a second externally in the mask); pressure sensors (headrest),
linear potentiometer placed on the spring inside the mannequin.
The signals were converted by a PMD 1208 LS, Measurement Computing
Systems analog-to-digital converter. All software was written in C++. A Laerdal
Medical Little Annie mannequin portrayed the victim for each responder. The
mouthpiece was exchanged and discarded for every responder while the bacteria filter
and mannequin “lungs” were replaced every five participants or fewer as needed.
Figure 3. Components of the device.
#�������#�������#�������#�����������Training sessions were offered at various times and locations for a period of 90
days. Participants were blinded to the type of training they would undergo. After the
group training session, each individual was to schedule a testing date a minimum of
fourteen days to maximum of twenty-one days after training. Participants’ self selection
of training and experimental times contributed to the randomization process.
Upon returning for the experimental session the experimenter ensured consent
forms were completed, verified training, and read the participant the appropriate
instructions. All were informed that they would enter a room with an unconscious victim
(no breathing, no pulse). Their task was to perform CPR until help arrived (the victim
would remain unconsciousness for the entire scenario).
The groups not utilizing the device were informed that “tools” would be available
adjacent to the victim. These included a mask for ventilating the victim as well as pads
associated with the defibrillator. The components were identical to those available in the
device condition.
The device group had the added benefit of the headrest and of course, the
protocol, audio and visual cues and feedback presented by the device. Pre-experiment
instructions encouraged device-group participants “…to follow the device as closely as
possible”. A still frame of the video instruction to place the headrest is displayed in
Figure 4.
Figure 4. Snapshot of video instruction "place headrest"
After completing the scenario, the researcher praised the efforts of the participants
and reassured them it was simply a simulation with an inanimate object. The debriefing
also included a video-cued recall exercise as well as survey to assess actions. They were
finally thanked for their time and paid $30.
RESULTS Sensors placed inside the mannequin collected the clinically relevant data.
Variables appraising CPR performance received highest priority. These generally
include measures relating to ventilation and circulation. These types performance
parameters can serve as inputs to models of oxygen delivery and ultimately impact
survivability.
Data supporting the COCOM-based interests were captured on video and assessed
through several coding procedures. Four research assistants, blind to hypotheses,
observed behaviors and selected corresponding action descriptions. The researchers then
compiled those data and determined the appropriate control mode.
Lastly, survey instruments allowed the participants to provide their perceptions,
comments and display their knowledge. The survey was utilized to provide additional
support for the behavioral data.
Throughout the rest of this work, participant groups are designated (as in Figure
1) by the training-support convention. The GRE-NO group received GRE training and
was not supported by the device. Participants in the group CPR-DEV received training in
CPR and utilized the device during the testing phase of the experiment.
#�������������%���#�������������%���#�������������%���#�������������%������� The availability of the physiological data for each group is itself informative.
The first data that captured the researchers’ attention were the number of responders
registering values for the dependent physiological measures (the “n” for each group).
Each sensor had a minimum threshold in order to distinguish a valid input from readings
attributed to other artifacts. For example, a 50mL threshold was set for the flow meter to
measure breaths. The threshold was set because small rescue breaths would be
indiscernible from the air movement induced by chest compressions. The 50mL
minimum enabled the researchers to identify deliberate rescue breaths in the sensor data.
As stated, 20 participants composed each group. For GRE-NO, only 6 participants
provided measurable breaths and compressions (n = 6). In contrast, the remaining groups
preserved a minimum of seventeen or 85% (GRE-DEV). Only 30% of the participants in
GRE-NO were able to meet the minimum performance requirements to surpass sensor
thresholds. However, this does not suggest those six in GRE-NO performed well; it only
states their performance was discernible from no action at all (unlike the other 14
members of their group). The abysmal performance of GRE-NO supports the first
performance hypothesis [CPR1] which predicted GRE-NO would perform far worse than
the other groups. The difference between the number of participants achieving
measurable performance for GRE-NO and GRE-DEV (equivalent training, but GRE-DEV
used the device) proved significant (Fisher’s exact test, one-tailed, p < 0.001).
Table 1 provides summary performance measures for critical CPR variables by
group. The mean for each group is displayed with the standard deviation in parentheses
beneath the mean. The researchers employed a one-way multivariate analysis of variance
(MANOVA) to calculate differences attributable to the groups. The results indicate a
significant difference in performance measures for group among all measures (Wilk’s
Lambda = 0.126, approximate F(20,279.5) = 12.1; p<0.001).
Corresponding univariate analyses and post-hoc, pair-wise testing is also
presented where appropriate. Scheffe tests were selected for all post-hoc testing. Despite
results from Levene’s test for equality of variance (p < 0.002) for 4 of 5 variables which
would advocate another test such as Dunnett’s T3, Scheffe’s test proved to be the most
conservative for avoiding Type I errors. Statistical results are posted in Appendix F.
Significance is determined at the α = 0.05 level.
Table 1. Group means for CPR performance variables. (Standard deviations in parentheses below).
Group
Breath : chest
compression (CC) ratio
Inspired volume with each
breath (mL)
Chest compression (CC)
frequency (per minute)
Chest compression
(CC) depth (in.)
GRE-NO
0.7 : 4.8 (1.2 : 4.7)
125 (204)
43.4 (31.0)
1.2 (0.8)
GRE-DEV
2.7 : 14.7 (0.9 : 4.6)
982 (383)
75.7 (21.8)
1.4 (0.3)
CPR-NO
1.9 : 13.2 (0.7 : 2.4)
460 (316)
95.5 (10.5)
1.8 (0.3)
CPR-DEV
2.6 : 14.3 (0.6 : 2.4)
867 (367)
84.1 (12.2)
1.6 (0.3)
DEV-DEV
2.3: 15.1 (0.5 : 2.4)
1235 (423)
91.1 (5.2)
1.8 (0.2)
The first physiological hypothesis [CPR 1] predicted GRE-NO would severely
underperform compared to the other groups and provide clinically meaningless
interventions. Univariate analyses for all five dependent measures resulted in significant
differences driven largely by the GRE-NO data. The univariate results were: breaths
administered (F(4,88) = 17.3, p < 0.001), chest compressions delivered (F(4,88) = 28.1, p
< 0.001), chest compression rate (per minute) (F(4,88) = 24.0, p < 0.001), compression
depth (inches) (F(4,88) = 4.9, p < 0.001), and average volume (mL) delivered with each
breath (F(4,88) = 30.9, p < 0.001). Post-hoc testing revealed GRE-NO means were
significantly lower than the other groups in the number of breaths and compressions
delivered and the rate of compressions (p < 0.003). Further, their breath volume (M =
125, SD=204 mL) and chest compression rate (M = 43, SD =31 per min) were well
below the AHA guideline of 800 – 1200mL per breath and 100 compressions per minute
respectively.
The second physiological prediction [CPR 2] forecast a slow start for GRE-DEV
but insisted their performance would quickly improve and even surpass CPR-NO. Figure
5 shows group chest compression rate means for each successive cycle. Following GRE-
DEV’s progress from their first cycle to their last shows an improvement in rate from the
low sixties to the high eighties. Similar gains are portrayed in Figure 6 charting breath
volumes. Their first cycle falls short of the recommended 800mL. However on the next
cycle they exceed the 800mL minimum and deliver sufficient breaths in all remaining
cycles for a mean breath volume of 982 (SD=383) mL. GRE-DEV significantly
surpassed CPR-NO (M = 460, SD = 316 mL) in delivering breath volumes (p=0.001)
lending credence to the hypothesis that GRE-DEV could outperform CPR-NO in some
instances.
A multivariate test produced a significant difference for the slopes (Wilks’
Lambda = 0.566, F(8,188) = 7.7; p<0.001) as did the univariate tests for chest
compression rate (F(4,92) = 9.0, p<0.001) and average breath volume (F(4,92) = 9.4,
p<0.001). Post-hoc Scheffe tests revealed GRE-DEV showed significantly improved
chest compression rate over GRE-NO, CPR-NO and DEV-DEV (p ≤ 0.006). Similarly,
GRE-DEV exceeded the learning rate of GRE-NO and CPR-NO for average breath
volume (p ≤ 0.001).
The third performance hypothesis [CPR 3] focused on CPR-NO and assumed they
would likely perform adequately, but were also subject to a higher probability of error.
Overall, their performance adhered to guidelines and demonstrated proficiency but
proved no better than the other device groups. CPR-NO’s means were not significantly
better except in the case of exceeding GRE-DEV for chest compression rate (p=0.042).
However, as predicted, they did demonstrate inferior performance in one aspect. Their
mean breath volumes (M = 460, SD = 316 mL) not only failed to meet the minimum
suggested volume (800mL), but significantly lagged all device groups in this
performance measure (p < 0.017). Clinically, this performance would do little to deliver
oxygen to the blood stream.
The AHA recommends a volume of 800mL to 1200mL for each breath. This
range accounts for the roughly 400mL of “dead space” that must be cleared between the
victim’s mouth and alveoli. This dead space volume must be completely purged to
optimize oxygen transfer to the blood stream. In essence their optimal chest
compressions are wasted; their efforts would result in circulating deoxygenated blood.
Mean CC frequency per cycle
30
40
50
60
70
80
90
100
110
1 2 3 4 5 6 7 8 9 10 11 12
cycle
CC
per
min
ute
gre-no
gre-dev
cpr-no
cpr-dev
dev-dev
����
����� � ���
� �
��� ���������
����� ����
Figure 5. Group means of chest compression rate for each cycle
The last physiological hypothesis [CPR 4] conjectured both trained groups
supported by the device (CPR-DEV, DEV-DEV) would outperform the other groups and
closely adhere to protocol prescribed values. Table 1 shows means for all five variables
closely track the prescribed values in the AHA protocol for both CPR-DEV and DEV-
DEV. However, post-hoc tests revealed these two groups only outperformed GRE-NO in
all five dependent measures (p < 0.001) with the exception of the difference in chest
compression depth means for GRE-NO and CPR-DEV (p = 0.20). Neither of these
trained groups with the device surpassed GRE-DEV on any variable ( p > 0.18)
Avg Breath Vol (mL) per cycle
0
200
400
600
800
1000
1200
1400
1600
1 2 3 4 5 6 7 8 9 10 11 12cycle
Bre
ath
Vol
(mL)
gre - no
gre - dev
cpr - no
cpr - dev
dev - dev
����
����� � ���
� ��� � ��
����������
Figure 6. Group means of inspired volume per breath for each cycle
"*"*+ �%���"*"*+ �%���"*"*+ �%���"*"*+ �%��������������$����
Four blinded coders performed the control mode analyses by reviewing video and
recording observations to a data template. The coders provided COCOM based for
subtask actions executed by the participants. Inter-rater reliability was assessed with
Cohen’s Kappa (Kappa = 0.899, see Appendix F). This tabulation was well above the
commonly accepted 0.70 level threshold validating the use of the coding results.
Table 2 summarizes the proportion of actions by mode for each group. A
multivariate analysis of the control mode data resulted in a significant difference for
group (Wilks’Lambda = 0.049, approximate F(12, 246.3) = 43.6, p < 0.001). The
appropriate univariate and post-hoc Scheffe results are discussed below.
Table 2. Group means for COCOM classification (in percent). (Standard deviations in parentheses below).
Group %Scrambled %Opportunistic %Tactical
GRE-NO 90
(12)
4
(7)
5
(7)
GRE-DEV 30
(15)
66
(15)
4
(8)
CPR-NO 19
(22)
16
(7)
64
(25)
CPR-DEV 17
(12)
62
(16)
21
(17)
DEV-DEV 15
(9)
60
(10)
24
(12)
The COCOM data conform well to the behavioral hypotheses. Haphazard
responses, trial and error approaches contributing to scant comprehension of the process
constituted the first behavioral hypothesis [BEH 1]. GRE-NO responders demonstrated
their lack of knowledge and an inability to improve during the scenario as all twenty
participants were classified in the scrambled control mode. Almost all of their actions (M
= 90%, SD = 12%) epitomized scrambled behavior. Post-hoc tests revealed the 90%
mean for GRE-NO was significantly higher compared to scrambled scores for all other
groups (p < 0.001).
In contrast, GRE-DEV benefited from the provisions of plans, cues, and
feedback and demonstrated an opportunistic control mode affirming the second
behavioral hypotheses [BEH2] which predicted the device would elevate their level of
control. GRE-DEV posted the highest proportion of opportunistic behaviors (M = 66%,
SD =15%). Post-hoc tests showed this percentage was significantly higher than the two
non-device groups (p < 0.001) but not statistically different from the other two device
groups (p > 0.658). Most of GRE-DEV ‘s remaining actions (M = 30%, SD = 15%)
resembled scrambled behavior. Only two participants (10%) in the group coded as
scrambled while the rest convincingly performed in an opportunistic mode.
The third behavioral prediction [BEH3] involved the CPR-NO group. Well
trained but unsupported (no plan, cues, or feedback), this group relied on knowledge
retrieval to perform the task. Adequate performance under such conditions demanded a
tactical control mode; and the data concur. Tactical actions accounted for the majority
(M = 64%, SD = 25%) of their activity, significantly higher than any other group (p <
0.001). Their opportunistic score (M = 16%, SD = 7%) was significantly lower than the
three device groups (p < 0.001) CPR-NO’s scrambled score (M = 19%, SD = 22%) was
comparable to the three device groups and reveals that participants in all groups made
errors. Of the twenty in CPR-NO, eighteen coded as tactical and only two scrambled
confirming the control mode prediction.
The final behavioral prediction [BEH4] conceded that despite their training, the
presence of the device would engender an opportunistic mode of control for CPR-DEV
and DEV-DEV. The device captivated these participants. CPR-DEV tallied 16 members
in opportunistic, 3 in tactical and 1 in scrambled. Again, the three device groups did not
differ significantly in their opportunistic scores (p > 0.658), nor scrambled scores (p >
0.074 ), but GRE-DEV did lag the other two groups in tactical actions (p < 0.026). All 20
participants in DEV-DEV registered opportunistic modes of control.
#��������)���������#��������)���������#��������)���������#��������)�������������A total of nine subtasks were identified as necessary steps to comply with the
AHA protocol. The researchers recorded the number of correctly completed steps
achieved by each participant. The analysis was confined to only the first cycle since all
tasks must be completed the first time through the procedure. For example, if the pads
are placed in the first cycle, there is no need to complete that step in each cycle.
Subsequently, only a subset of the tasks was repeated (Appendix C, Error Coding,
contains the list of subtasks).
Two coders independently evaluated each action on the video and determined if
participants correctly completed the required action. Of the 900 judgments, only 19
discrepancies emerged resulting in high concurrence between the raters (agreement
exceeded 97.8%). Table 3 contains the number of protocol steps each group performed
correctly.
Table 3. Mean number of protocol steps executed correctly (9 max).
Protocol Steps GRE-NO GRE-
DEV CPR-NO CPR-DEV DEV-DEV
MEAN 1.6 7.80 7.85 8.45 8.65
SD 1.46 1.05 1.49 0.88 0.49
These group means did demonstrate a statistically significant difference (F(4,95)
=135, p<0.001). The results were congruent with all behavior hypotheses stated earlier.
GRE-NO was unable to accomplish most subtasks critical to CPR and defibrillation. Post-
hoc tests demonstrated GRE-NO was statistically disparate from all other groups (p <
0.001). The second [CPR 2] and fourth [CPR 4] hypotheses predicted the device groups
would complete all subtasks and the data here support that. The data also showed CPR-
NO [CPR 3] demonstrated sufficient recall of the procedure as they too regularly
completed all steps.
The completion of protocol subtasks was also assessed for proper sequencing.
The scrutiny of sequence proved formidable for CPR-NO. As postulated in the third
performance hypothesis [CPR 3], relying on memory proved onerous. CPR-NO
demonstrated problems recalling the correct sequence. While they remembered the
necessary steps, 60% of the responders failed to perform them in the correct order. Table
4 displays the number of responders in each group that followed the sequence correctly
(Yes), and those that failed to do so (No).
Table 4. Number of responders exhibiting correct sequencing of subtasks.
Correct Sequence
all-DEV (GRE-DEV, CPR-DEV,
DEV-DEV)
all NO-DEV (GRE-NO, CPR-
NO)
CPR-NO
GRE-DEV
Yes 45 8 8 15
No 15 32 12 5
Fisher’s Exact test was again used to determine statistical significance for
multiple 2 x 2 comparisons. The three device groups significantly outperformed both no-
device groups (all-DEV vs all NO-DEV, p < 0.001). Removing GRE-NO from the
analysis to unburden CPR-NO with disproportionate inefficacy still yielded a significant
difference (all-DEV vs all CPR-NO, p < 0.001).
��,���%�����,���%�����,���%�����,���%�������
Survey responses resided on a 10 point scale and were collected post-experiment.
Inquiries coincided with COCOM parameters such as the availability of plans, the
utilization of cues and feedback, and the ability to comprehend the situation. The
researchers designed the low, middle, and high values of the scale to embody scrambled,
opportunistic, and tactical modes of control, respectively (see Appendix C for the Post-
Run Questionnaire Survey as well as Appendix F for full statistical analyses). The
survey was intended to provide additional support for the COCOM predictions.
A multivariate analysis examining the groups for all survey variables proved
significant (Wilks’ Lambda = 0.047, approximate F(32, 326.1) = 13.2, p<0.001).
Additional statistical analyses are given where appropriate.
������The first survey item attempted to reveal the driving force initiating responder
actions. Lower survey scores indicated a random approach, the middle range suggested
actions were guided largely by the system, and higher numbers indicated the use of a
retrieved plan, consistent with the COCOM continuum. A significant result between
group means (Table 5) was confirmed (F(4,95) =12.0, p<.001). Post-hoc Scheffe tests
revealed that CPR-NO did differ significantly from the other groups (p ≤ 0.029) with the
exception of DEV-DEV.
Table 5. "Action" survey results
The means for each group are in line with the outlined behavioral hypotheses. In
particular, CPR-NO turned in the highest rating (M=8.25, SD=1.1) suggesting the use of
memorized plans (no system cues were available in this condition) [BEH3]. GRE-DEV
registered a mid-range score (M = 5.75, SD=1.3) confirming their leveraging of system
functionality [BEH4]. GRE-NO’s score of 6.05 is not consistent with the first behavioral
hypothesis [BEH1], nor is it likely to be an accurate report as cue availability was
Action Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV
MEAN 6.05 5.75 8.25 6.75 7.85
SD 1.66 1.29 1.11 1.74 1.08
essentially non-existent. Perhaps they believed the presence of tools (such as mask and
pads) served as cues, but their proclivity to forgo tool use is further behavioral evidence
contradicting their survey mean.
%������&�The second item on the survey captured the participants’ perception of the
importance of feedback to their performance. Higher ratings indicated that feedback was
utilized and fostered ample improvement. The middle of the scale suggested occasional
use and limited enhancement, and the low end of the scale represented the absence of
feedback. In addition to collecting participant sentiment, it also served as a check – the
paltry feedback provided in the “no device” conditions left little information to be
gleaned (though trained rescuers astutely claimed that leaking breath sounds constituted
feedback).
Again, a significant difference was observed between the groups (F(4,95) = 31.7,
p<0.001). As feedback is a critical element in JITS, behavioral hypotheses posited the
device groups would rate feedback favorably whereas the groups without the device
should not. GRE-NO and CPR-NO had mean scores of 2.85 and below while the three
groups with the device tallied survey scores between 7.0 and 7.8 (see Table 6).
Table 6. "Feedback" survey results
Feedback Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV
MEAN 2.85 7.80 2.45 7.00 7.60
SD 2.15 1.85 2.39 1.89 2.21
Post-hoc tests revealed that device groups did not differ significantly from each
other, nor did the no-device groups, but each device group differed significantly from
each no-device group (all p-values <0.001).
An adjuvant aspect of the feedback data was interesting to note. The researchers
tallied the number of participants receiving chest compression feedback in each group.
Classifying a participant as having received feedback required three or more instances
over the course of the scenario (participants receiving at least 3 feedback occurrences
generally encountered many more instances of feedback where as the researchers
determined 1 or 2 instances was not sufficient to justify the feedback categorization).
Table 7 displays the sums.
Table 7. Number of participants provided feedback by the system.
Feedback Provided GRE-DEV CPR-DEV DEV-DEV
Yes 16 8 15
No 4 12 5
Fisher’s Exact Test calculated the differences between CPR-DEV and DEV-DEV
resulting in a significant difference (one-tail p =0.026). This finding complied with
behavioral hypotheses quite well as JITS reduced the emphasis on retention. One
explanation could be that those trained with the device believed they could depend on it
later – comforted by that feedback would optimize their performance. During training,
DEV-DEV may have learned to rely on the device for guidance and processed the stimuli
on a more superficial level. In contrast, CPR-DEV was not aware an assist device would
be present for the test and felt more compelled to remember their training.
��������Participants were asked to rate their comprehension of the progress and results of
the scenario. Those comfortable in their understanding and confident in their efforts
gravitated toward the higher values. In contrast, the lower end of the scale was crafted
for participants with no ability assess their performance or postulate the outcome. The
differences in means for this inquiry were statistically significant (F(4, 95) =9.1, p <
0.001). Those participants aided by the device at least felt they maintained some
understanding of the events in comparison to their counterparts without the device.
Comparing the group means (Table 8) revealed GRE-NO significantly lagged each device
group (p ≤ 0.001) however, CPR-NO did not (p ≥ 0.232).
Table 8. "Outcome" survey results.
Outcome Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV
MEAN 3.85 7.15 5.60 7.30 7.30
SD 2.34 1.95 2.18 2.55 2.17
DISCUSSION The vision of JITS is to develop systems capable of enhancing user performance
as they confront unfamiliar tasks requiring immediate intervention. JITS accomplishes
this challenge by providing adaptive plans, cues, and feedback tailored to the user’s
understanding. System flexibility is required to adapt to dynamic contextual conditions
as well as user behavior.
The results of this study validate the tenets of the JITS approach. The data
demonstrated that participants with no specialized knowledge, skills or training could
perform as well as those given training and the opportunity to practice. This was
achieved simply by providing novice participants a JITS device. Additionally, their
performance was in the range of clinical guidelines suggesting physiological benefits.
"# ����-������� ����"# ����-������� ����"# ����-������� ����"# ����-������� �������� The JITS benefits are most evident when examining the performance differences
between GRE-NO and GRE-DEV. All participants in these groups lacked declarative and
procedural knowledge required to perform CPR. All completed a training session in
which they learned strategies for the verbal section of the GRE which in no way prepared
them for the CPR task. The difference between the groups was their use of the device
during the experimental test; GRE-DEV reaped significant benefits from a JITS designed
device. The data affirmed the performance gains.
The untrained responders without the device failed to achieve anything of clinical
import. Only 30% of GRE-NO provided inspired volumes large enough to be detected.
They delivered approximately one-third of the chest compressions required by the AHA
guideline. Other performance measures such as volumes and compression frequency also
failed to attain clinical relevance. From the CPR viewpoint, no breaths were delivered to
the lungs and no oxygen delivered to the vital organs that need it. This finding was not
unexpected; untrained responders with no support are unable to perform CPR effectively.
However, this study measured their fecklessness, providing a means for quantitative
comparison.
Providing a similar population of naïve responders a JITS system resulted in
vastly improved performance. The device aided group successfully delivered life-saving
treatment despite having no previous training or knowledge. Their breath-to-
compression ratio approached the guideline thanks to the pacing of the device. The
breath volumes delivered tracked the middle of the recommended range because sensors
drove feedback to guide them there. While their compression depth and frequency were
slightly below threshold, their performance still provided physiological relevant
treatment.
In a real cardiac event, this intervention would take place during the transit time
of professional responders. This four to eight minute duration is often void of
resuscitation efforts. The data suggest novices may be capable of providing effective
life-saving measures during this interval when equipped with a JITS device even if
untrained. A JITS designed intervention has potential to greatly increase the dismal SCA
survival rates.
An even more intriguing comparison is that of GRE-DEV vs. CPR-NO. This
comparison essentially examined the difference between class training and real-time
training. The data demonstrated GRE-DEV was able to perform on par with the trained
group, and in the case of oxygen delivery, outperformed CPR-NO considerably. This
suggests the real-time system achieved results comparable to class training. Should these
findings continue to be upheld by future experimentation, a case could be made
challenging the classroom training paradigm. Instead of training a few responders that
may never be called upon to act, training would be given to people actually responding to
a cardiac event.
The comparison of CPR-DEV vs. DEV-DEV did not result in any discoveries of
note. The distinction of these groups was created to examine the effect of specific
training prior to device use. Having used the device in training produced no measurable
advantage in this study. Since both groups performed close to the protocol
recommendations, this could be the result of a ceiling effect. Additionally, it should be
noted that the introduction of the device to a group trained without it was not hampered
when required to use the device during the experiment. In fact, comparing this group to
CPR-NO it is evident that the device helped a great deal in delivering effective breaths
and did not induce negative transfer effects.
������������������������������������������������For most of the untrained responders this experiment provided their first attempt
at CPR. How were untrained responders able to achieve such impressive performance?
It is clear from the data the JITS device enabled an untrained group to perform
successfully. The equivalently prepared group without the device failed to accomplish
anything that could positively impact survival.
By observation it was evident that all GRE-trained participants entered the
scenario with trepidation. GRE-DEV respondents showed some relief when they realized
they would receive computerized support. The device first gave the instruction “touch
here to start”. The intention was to capture the user’s attention and direct it toward the
device. This simple instruction set the stage for the upcoming interaction. This first,
simple command prepared the user for the more complex information that followed. For
some the command was immediately effective, others required a few seconds to orient
themselves.
CPR-NO, though received training, did not deliver breaths adequately. Of course,
they were not supported with feedback and had few resources for identifying their
deficiency. Therfore, they were unable to perceive their error and continued to perform
poorly. Though they administered effective compressions, their failure to provide
sufficient ventilation rendered the compressions far less effective.
It is apparent the CPR-NO participants were unaware of their impotent
ventilations. An ability to assess the progress of the task is a critical element in
controlling any system. It is particularly difficult for non-experts in stressful situations
according to COCOM. For those cognizant enough to ascertain the problem, they found
it difficult to correct their mistakes.
Laypersons’ difficulty in delivering rescue breaths is so ubiquitous that clinical
researchers and professionals advocated revision of CPR guidelines. Many called for the
simplification of the process which included increasing the number of chest compressions
instead of wasting effort on ineffective rescue breaths (Cummins, & Hazinski, 2000,
Sanders & Ewy, 2005). Further, the interruption of chest compressions resulted in the
absence of circulatory stimulus for excessive periods (Valenzuela et al., 2005). Many
shared their conviction as reflected in the November 2005 revision of the CPR guidelines
(ILCOR 2005). The number of chest compressions recommended was raised to 30 from
15 essentially attenuating rescue breath efforts by default.
This significant change to the guideline highlights the disparity between a static,
predetermined approach versus a flexible, adaptive JITS system. The revision is based
on the assumption that all resuscitation needs will be better served by the new
parameters. A JITS solution could make the determination during the case, and perhaps
even during each cycle and adjust the response according to the needs of the victim. If
the first breaths were inadequate, the system would coach the responder to provide more
effective breaths. Processing the first breath inputs (with active sensors) might result in
feedback alerting the responder to a problem and offer corrective assistance such as
“press harder on mask” and “give 2 larger breaths”. Perhaps the first attempt at breaths
is unsuccessful and too much time has passed, so the system decides to move to chest
compressions. What if fifteen or twenty or twelve was the optimal number of
compressions to attempt before going back to breaths? Such intervention can only arise
from real-time systems capable of monitoring the context and designed to be sufficiently
adaptable. As demonstrated in the experiment, an adaptive, monitored plan, cues and
feedback loops provided a significant advantage to participants in the device groups.
An additional impediment to lay responder success is their poor retention of CPR
skills. Morgan, Donnelly, and Lester, (1996) surveyed the scope of the problem in
England and found that only 7% of trainees were able to perform safe, effective CPR six
months after training. Additional studies reveal the retention problem is not limited to
non-professionals. Medical students demonstrate similar retention failures (Graham &
Scollon, 2002, Fossel, Kiskaddon & Sternbach, 1983). Thus the performance delivered
by CPR-NO should be considered optimal since only two weeks passed between training
and test. Research findings predict skills would degrade considerably as more time
elapsed between training and test (Morgan, Donnelly, & Lester, 1996, Starr, 1998).
Memory should not be a significant impediment to healthy patient outcomes.
Even unintelligent systems can provide memory aids. On a surprise retest 30 days after
training, Star (1998) observed a 4-fold improvement in performance skills for participants
using a simple prompt system versus those relying on memory alone.
Evidence that a brief prompt was capable of improving performance should spark
a great interest in the development of a more complete JITS solution. The prompts were
bound to a sequence, did not benefit from contextual awareness and could not leverage
real-time data. A JITS system for CPR could enhance the response of trained responders
as well as enable sufficient performance from novices.
"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+���������� This research pioneered one of the earliest applications of the COCOM to
generate dependent measures sufficiently robust for quantitative analysis. Very few
researchers have attempted to marshal the COCOM predictions in such a manner largely
due to the inchoate bounds of control modes and an inability to derive cogent measures.
Stanton, Ashleigh, Roberts, and Xu, (2001) provided the first empirical support for
COCOM hypotheses. However, they’re coding process was not developed with the rigor
employed in this work. Ultimately, successful utilization of COCOM parameters
requires a very specific context. Therefore, previous efforts can serve only to guide
future development. The measures described in this work are not transferable.
The basis for the COCOM-derived dependent measures reside in Hollnagel’s
mode characteristics. Each of the subtasks (i.e. holding mask, placing pads, delivering
compressions) provided subtleties for control mode classification. For example, one
technique forholding the mask was the anesthesia-style hold taught in training. This grip
exemplified tactical control as it demonstrated knowledge retention and skill.
A participant portraying an opportunistic mode first showed evidence of perusing
the screen prior to placing the mask. Then a two-handed hold was employed with a
downward thrust near the nose and chin. This provides a logical opportunity to redesign
for the novice user. While there are benefits to the anesthesia hold, the designers
surmised this technique would be very difficult to portray and for naïve users to learn in
real-time. As a result, a compromise was reached. The conveyed information for mask
holding was simplified to a two-hand hold placing pressure in two points to create a seal.
This approach was far simpler to portray and perform. In most cases, the system
instructed the user to “press harder” during mask placement. A pressure sensor in the
cuff of the mask was set to ensure enough downward force was applied to generate a
good seal to ensure rescue breaths did not leak from the mask. A poor seal was a
significant problem for CPR-NO which did not have the sensor-driven feedback.
Scrambled control designations for the mask placement task took many forms.
Many of the GRE-NO participants never used, and some never noticed the mask was
available. Such instances received the scrambled classification. Two of the GRE-NO
responders attempting to use the mask inserted the mouthpiece into the mannequin and
blew into the mask. Reversed mask insertion ensured no ventilation and exemplified the
hallmark trial-and-error approach associated with scrambled control. Members of other
groups also recorded scrambled actions in the mask hold because they only used one
hand to seal the mask to the victim’s face. This practice always produced leaks and
therefore less effective breaths. Without the aid of the device or serendipitous recall from
training, this error usually went uncorrected.
Utilizing COCOM parameters and translating them into CPR-required subtasks
produced a matrix capable of capturing responder behavior. Though Hollnagel cautions
mode demarcation is inexact, the data conformed very well to hypotheses.
As predicted [BEH 1], GRE-NO performed 90% of their actions in scrambled
mode. All 20 participants coded as scrambled overall for the scenario. Based on the fact
that they had neither skills nor knowledge to employ, there was little probability they
could perform in any other manner.
CPR-NO demonstrated tactical control a majority of the time (64%). They too
conformed to hypotheses [BEH 3] as it was predicted they would make mistakes due to
their reliance on recall. Their erroneous behavior was highlighted by 20% of their
actions coded as scrambled and 60% of the responders failing to properly sequence the
subtasks. Despite great success in adhering to the protocol for breath-compression ratio
and compression frequency, mistakes such as their poor ventilations would adversely
affect patient outcomes.
The variance in CPR-NO means was also noteworthy. CPR-NO recorded the
highest standard deviations in Table 2 (SD = 22 scrambled, SD=25 tactical), indicating a
considerable range in the group performance. It is also important to note the experiment
took place only two weeks after their training and still substantial recall errors surfaced.
The variance suggested some responders displayed superb retention while severe decay
was evident in others. As other research has shown, performance of all members can be
expected to decline with greater delays between training and test. Over time, a reduction
in performance would be predicted as would a correlated reduction in variance.
Examining the device groups, GRE-DEV was hypothesized to demonstrate
opportunistic control and the data support that claim. Sixty-six percent of their actions
coded as opportunistic. They also made some mistakes as evidenced by 30% of their
actions classified as scrambled. Observation of their performance indicated that many of
the mistakes they made occurred early in the scenario. Anecdotal examples include some
confusion while searching for the mask, and giving compressions with the wrong grip
and compressing much more slowly than the required pace. Most device-aided
participants corrected their actions in the subsequent CPR cycles.
Trained groups using the device also demonstrated control profiles easily adapted
to the COCOM paradigm. At firs t glance, their training should have enabled them to
demonstrate some tactical mode characteristics. Nonetheless, the presence of the system
would likely drive their actions in the direction of opportunistic control. The question to
be determined empirically was the proportion of opportunistic to tactical methods. Each
trained-device group performed roughly 20% - 25% of their actions in tactical mode and
more than 60% in opportunistic. Two factors are most plausible for generating this
distribution. First, the system was explicitly designed to seize attention and engage the
operator throughout the scenario. Second, with what can be considered limited training
and practice, most operators would likely defer to the system for expertise.
The post-experimental questionnaire served as yet another instrument employed
to assess the usually amorphous bounds of control mode categories and provided
converging evidence for the hypotheses. The survey instrument utilized multiple scales
to represent different COCOM parameters. Specifically, scales represented action
selection, utilization of cues and feedback, determination of outcomes, for goal setting,
and time pressure. The inquiries concerning action drivers (includes cues), feedback
utilization, and outcome determination yielded results consistent with hypotheses
In general, GRE-NO exhibited a scrambled control mode as they reported no use
of feedback and minimal determination of the outcome. GRE-DEV showed a high
reliance on cues and feedback. CPR-NO ranked feedback very low and recorded the
highest action score avowing their use of a memorized plan to drive their actions. One
disappointment in the survey results came from the time pressure scale. According to the
COCOM, operators in scrambled mode have a proclivity to perceive an enormous limit of
the time available. However, due to the length of the scenario and lack of purposeful
activity, five minutes probably felt like a long time superseding any time pressure they
may have experienced at the start of the scenario (it did to the researchers watching them
struggle).
Attempts to impugn the validity of the COCOM measures must contend with
converging evidence from the coding and the survey data. The COCOM proved valuable
not only in predicting human behavior and providing design guidance, but also as a
measurement tool to evaluate the system. Though it may be impossible to describe
global COCOM measures, a well defined context not only affords their use, but benefits
from their instantiation.
"���������"���������"���������"�������������Could JITS replace traditional class training? There are many advantages to
widespread deployment of such a system. The delay in getting professional responders to
the scene is well documented. Precious moments are lost in transport. Additionally, it
has been shown that even when they arrive, they are not performing optimally.
The most obvious benefit is the augmented number of available responders and
thus the ability to initiate life-saving treatment immediately. The paltry percentage of
trained citizens currently limits the probability that a witness to SCA will be prepared to
act. The limited response incurred is directly attributable to the current, flawed training
paradigm. Instead of training large groups of people that may never apply their skill,
JITS methods can train the few people that need to perform BLS immediately.
Good Samaritans, previously helpless because they lacked 6-12 hours of training,
would now be empowered to act. Herein resides the greatest decrease in response time
and greatest clinical benefit to the victim. When a witness can immediately respond to a
cardiac event effectively, the victim’s probability for a healthy outcome is vastly
improved. The researchers believe the JITS framework could be used to transform any
willing bystander into a capable responder.
)������ �There are several notable limitations to this study. The most apt concession
acknowledges the difficulty in mapping statistical significance on to clinical relevance.
While statistical differences surfaced for many variables, the clinical relevance of these
differences is not always evident. Conversely, differences that failed to breach statistical
thresholds could belie substantial medical implications. For example, are clinical
benefits realized when the mean volume per breath rises from 867mL to 962mL? This
work cannot answer that question and previous work with models has yet to provide a
satisfactory resolution or even determine if the question is worth asking.
A second limitation indicts the device itself. The criticality of response time
cannot be overemphasized. Yet, utilizing the JITS solution did cost the responders time
compared to the no-device groups. Appropriate usage and placement of tools, processing
lags to read sensors and generate feedback imposed delays beyond the control of the
responder. The algorithms contained “hold points” to ensure prerequisite steps were
performed prior to proceeding. These potentially can stall the procedure and thus
requires prudent threshold setting. Self-paced, CPR-NO completed more cycles per
scenario than any other group and spent less time between cycles demonstrating some
time cost attributed to using the system.
A more general limitation for JITS resides in its technological dependencies.
Algorithm and sensor development could present significant challenges and have
ramifications for human performance issues. For example, if the task analysis requests
unattainable data, proxy variables, or modeled data, procedure modifications may be
necessary. Relying heavily on technology also demands prepared contingencies for
system faults and failures.
The researchers and development team encountered several examples of
technological obstacles. The available computational processing speed demanded our
first concession. A potentiometer in the mannequin’s chest provided feedback for chest
compressions. However, the signal delay coupled with the algorithm made it impossible
to assess performance and deliver feedback during the cycle. Thus, the feedback, based
on the previous cycle, was presented to the responder at the beginning of the next cycle.
In a sense, the user was required to “remember” the force and speed of the previous
performance and adjust accordingly. It is not clear that the participants detected this
discrepancy and the concern is further assuaged by the improved performance due to
feedback. Nonetheless, future JITS systems will face numerous technological challenges.
Sensor placement inside the mannequin presented a further technical impediment.
Obviously, this would be impossible for a system intended for human use. Placement
inside the mannequin was performed for two reasons. First, it provided the most reliable
means of data collection – critical for experimentation. Second, these measures drive the
design of the sensors needed for the next iteration of the CPR device. In one case,
engineers are leveraging flow meter, inspired volume, and leak data in order to establish a
model capable of assessing inspired volume without an internal sensor.
Beyond technology, a further constraint is the need for tasks to be well structured.
The nascent stages of JITS are ill equipped to address the perplexing issues connate with
indeterminate tasks. The application of JITS requires a certain level of predictability.
Fortunately, a rigidly prescribed BLS protocol provided the structure necessary to
constrain and therefore predict most operator actions. This procedure relies on few
actions and subtasks. More complex activities will demand a great deal of contingency
planning and anticipation of off-normal conditions and actions.
Finally, the human limitations must be considered. For example, no amount of
JITS will enable recreational joggers to run 100 meters in 9.78 seconds (a physical
limitation). With tasks requiring highly practiced skills, such as landing an aircraft, JITS
will not mitigate a novice’s oscillatory control during a first attempt (automatized skills).
Nor can it be expected that a JITS system can equip a Private First Class with the war-
gaming prowess of a General (experience/knowledge).
In addition to motor skills, the emotional stability of the responder must also be
considered given the gravity of this task. There was no way to simulate the stress that
would likely envelope responders dealing with a real victim. For the most part,
participants maintained an emotional level that did not significantly degrade their
performance. However, this finding should not be generalized to a scenario involving a
real life. More experimentation is needed.
However, if designers are mindful of the capabilities and applicability of JITS
systems, ample domains can benefit from such support. The present demand for structure
does not nullify the applicability and benefits of JITS. Countless tasks are governed by
such protocols and can be seen in many domains including: aviation, aerospace, defense,
maintenance, manufacturing, medicine, process control, quality assessment, and safety.
������������������ The instantiation of the JITS framework can aid designers in the development of
systems crafted to improve non-expert performance of critical tasks. The need for such
support continues to grow as the demand for expertise outpaces supply. The JITS
framework can be employed in system development to attenuate specific expertise voids
supporting a wide range of objectives.
One issue that can be assuaged by JITS is user performance of infrequent tasks.
These atypical occurrences may include emergency situations or far less critical events.
The operator may have received training for the task, but the reality is the task is seldom
performed and the user is unpracticed. In this case, designers are dealing with a known
population. This enables some domain knowledge to be infused in the plan, cues and
feedback. The system must address the lack of practice, but probably does not have to
assume complete ignorance on the part of the user. It may be found that domain specific
cues and leveraging the knowledge of the operators can elevate performance.
Employing JITS systems aiding infrequent tasks could reduce or eliminate the
need for expending resources on training. Training for rare events is often ineffective
because of an extensive delay between training and practice and the misperception that
the training will never be needed. Additionally, these resources usually must be drawn
upon repeatedly not only for new users, but also refresher training. A JITS solution could
engender superior performance while curtailing wasted training costs.
The real power in broad deployment of JITS systems is the ability to significantly
increase the population capable of completing novel tasks. The system stores the
knowledge requirements, collects and interprets the data, and guides the user through a
workable plan. In situations formerly requiring significant expertise, JITS can supplant
that need and enable non-experts to complete the tasks.
A mature JITS system may do far more than elevate novice performance. It is
envisioned as a tool that would work with the individual from the first day of training and
accompany the user on every task mission. A robust student model would be constructed
over that time which would formulate optimized plans, cues, and feedback tailored to the
user. The same system would also recognize an unfamiliar operator and quickly diagnose
skill level. A mature JITS system is the manifestation of an intelligent, highly adaptive
system capable of supporting a diverse population of users accomplish tasks beyond their
proficiency.
APPENDICIES
)������.�)!�&/��'����"����)������.�)!�&/��'����"����)������.�)!�&/��'����"����)������.�)!�&/��'����"��������
)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"���������������
Mask no 2 hand - 2-hand seal 2-hnd anesthesia seal
breaths audible leak give when system
commands look for chestrise while
giving
cc 1handed; wrong grip; cc w beeps 15 consistent (despite
beeps)
body excessive move (waist
shoulders) looks @ screen little movement to
accomplish
Verbal what/how to do; express
frustration "Oh" count out loud; verbalize
plan
Start/Stop begin action- w/out finish
(correct) syst interrupt; follow sys ignore system to perform
correct tools fail to use; explore; search; used on command use correct w/out prompt can't find immediately anticipate
pads don't use watch screen to place place w/out looking
screen correct b4/w/out cue/fb place words up Mask no 2 hand - 2-hand seal 2-hnd anesthesia seal
breaths audible leak give when system
commands look for chestrise while
giving
cc 1handed; wrong grip; cc w beeps 15 consistent (despite
beeps)
body excessive move (waist
shoulders) looks @ screen little movement to
accomplish
Verbal what/how to do; express
frustration "Oh" count out loud; verbalize
plan
Start/Stop begin action- w/out finish
(correct) syst interrupt; follow sysignore system to perform
correct tools fail to use; explore; search; used on commanduse correct w/out prompt can't find immediately anticipate
pads don't use watch screen to placeplace w/out looking
screen correct b4/w/out cue/fb
)������.�"�!�)������.�"�!�)������.�"�!�)������.�"�!��.���� ������%����'�� ��.���� ������%����'�� ��.���� ������%����'�� ��.���� ������%����'�� �����
In-situ Error Coding Subject # ___________ Condition_______________ Date of Experiment____________ Errors during additional cycles Headrest Y N Mask used Y N Mask placement Y N 2 breaths Y N Shirt removed Y N Pads used Y N Tabs removed Y N Pad placement Y N Hands correct Y N Number of compressions ____________________________________
Post Run Questionnaire Subject # ___________ Condition_______________ Date of Experiment____________ Circle the ONE number that best represents your feelings during the experiment (do not circle multiple numbers nor place a circle between numbers). The text represents sentiments correlated with both ends and the center of the scale. 1. My actions during the experiment were driven by:
1A. Briefly describe any external cues or memorized plans (if used). 2. What best describes your utilization of feedback (FB) during the experiment?
2A. Please list specific examples of the feedback you employed: 3. What best describes your goal setting during the experiment?
1 4 2 3 5 10 6 7 8 9
1 4 2 3 5 10 6 7 8 9
1 4 2 3 5 10 6 7 8 9
Trial & Error Things I observed Memorized Plan
I didn’t get FB
FB helped a little
FB improved many actions
I focused on both CPR and the memorization task
I did not consciously consider goals
4. Describe the time pressure you felt while performing the task
5. How many minutes passed from the start of the scenario until you were told to quit ?
6. Describe your ability to understand what was occurring as a result of your actions.
7. Two lists of words were on the front of the victim’s shirt. Write as many
words as you remember.
8. What was the category that could describe most of the words? ____________ 9. What word was written in red? _______________
1 4 2 3 5 10 6 7 8 9
1 4 2 3 5 10 6 7 8 9
1 4 2 3 5 10 6 7 8 9
I focused solely on the CPR goal
Enormous time pressure
Pressured, but just enough time
Time wasn’t a factor
1 min 10 min
I had little understanding of the results of my actions
I could sometimes see and understand what was resulting
The results of all actions were clear to me
5 min
Video-cued Recall Form
Subject # ___________ Condition_______________ Date of Experiment____________ GOTO: first walking in room. Stop video as they approach victim/box.
Q. As you approached the victim, what were your thoughts? Describe the actions you were planning to take.
GOTO: point where approach patient, see and remove shirt.
Q. Did you see the words on the shirt and think about the memorization task? Did you consciously make a decision about studying the list or moving on?
GOTO: transition between breaths and compressions (1st cycle)
Q. Why did you move to chest compressions at this point? GOTO: last compression of the second cycle of compressions.
Q. Did you think about your future actions at this point? What were your plans?
)������.�%�!�����������0�����������)������.�%�!�����������0�����������)������.�%�!�����������0�����������)������.�%�!�����������0���������������
Participant Questionnaire Name ________________________________ phone __________________ Gender M / F Age _________ Year in school __________ Major __________ Cardiopulmonary Resuscitation (CPR) Have you ever been trained in CPR ? YES NO
month/year of initial training ___________ mo/yr. most recent training ________ Have you ever been certified in CPR YES NO Are you currently certified (AHA, Am Red Cross) in CPR ? YES NO Automated External Defibrillator(AED) Have you ever been trained to use an AED? YES NO If yes, when were you most recently trained?________________ The questions below are designed to assess your expertise. Most participants should not be able to answer these questions. If you don’t know the answer, simply place an “X” on the line. What are the ABCs of CPR? ________________________________________________ Briefly describe the purpose and technique in performing a “jaw thrust” ________________________________________________________________________________________________________________________________________________________________________________________________________________________ Briefly describe V-fib _____________________________________________________ _______________________________________________________________________
)������.��!�������������)������.��!�������������)������.��!�������������)������.��!�����������������
Instructions JITS w/Device Groups
Thank you for participating in this study. Today you will perform Cardiopulmonary Resuscitation (CPR) to the best of your ability on a mannequin designed for CPR training. I have your consent form. You have the option to withdraw from this experiment now, or at any time during the procedure. Should you wish to withdraw during the experiment, simply tell the researchers in the room you would like to stop. When the scenario begins you will find a mannequin representing the victim. The victim requires CPR. There is no need to evaluate the victim for responsiveness, as we have determined for you that the patient is unconscious and non-responsive. Assume a call to 911 has already been placed and paramedics will be arriving shortly to relieve you. Continue to perform CPR until you are told to stop. When you approach the victim, you will notice a white box with a red cross. This box contains supplies that you may use to perform your task. In addition to the supplies you will be using, the box also houses a video screen that will provide animated instructions and real-time feedback. Simply touch the screen to begin and the system will walk you through the CPR procedure step-by-step. The system is able to monitor your actions and offer feedback to help improve your performance. We encourage you to follow instructions as closely as possible. Please perform all tasks as if this was a real victim. The components that you will come in contact with are sterilized and/or replaced for each participant. Therefore, there is no need to mock any actions, or say you “would” perform an action, simply perform the action fully on the mannequin. Lastly, you may encounter a list of words at sometime during the procedure. We emphasize here that we want you to devote all your efforts to perform CPR to the best of your ability. CPR is very demanding, and we don’t expect participants to have much success memorizing the list. However, if you feel you can, try to memorize as many words as possible. That concludes your prebriefing instructions. Do you have any questions? Once you start, no one can answer any questions for you.
JITS JITS
Instructions Non JITS Groups
Thank you for participating in this study. Today you will perform Cardiopulmonary Resuscitation (CPR) to the best of your ability on a mannequin designed for CPR training. I have your consent form. You have the option to withdraw from this experiment now, or at any time during the procedure. Should you wish to withdraw during the experiment, simply tell the researchers in the room you would like to stop. When the scenario begins you will find a mannequin representing the victim. The victim requires CPR. There is no need to evaluate the victim for responsiveness, as we have determined for you that the patient is unconscious and non-responsive. Assume a call to 911 has already been placed and paramedics will be arriving shortly to relieve you. Continue to perform CPR until you are told to stop. When you approach the victim, you will notice a white box with a red cross. This box contains supplies that you may use to perform your task. Please perform all tasks as if this was a real victim. The components that you will come in contact with are sterilized and/or replaced for each participant. Therefore, there is no need to mock any actions, or say you “would” perform an action, simply perform the action fully on the mannequin. Lastly, you may encounter a list of words at sometime during the procedure. We emphasize here that we want you to devote all your efforts to perform CPR to the best of your ability. CPR is very demanding, and we don’t expect participants to have much success memorizing the list. However, if you feel you can, try to memorize as many words as possible. That concludes your prebriefing instructions. Do you have any questions? Once you start, no one can answer any questions for you.
NON JITS NON JITS
)������.�'�!�������������"����������)������.�'�!�������������"����������)������.�'�!�������������"����������)������.�'�!�������������"��������������
% "�* ��#������ �+����������,�-�Chi Square is not appropriate for 2 x 2 contingency table and also has limitations with small cell sizes (n < 5). Therefore, Fishers Exact test was utilized. Class 1 represents GRE-DEV while class 2 = GRE-NO.
Gp1 is the number of subjects able to register non-zero values for the physiological data. Gp 2 is the number of participants unable to perform above the noise threshold of the sensors. Therefore, performance is equivalent to no action at all.
The small p-value (0.0005 one-tail, 0.001 two-tail) indicates a significant difference in the distribution of these groups.
��������� ��� ��� �
�������� 17�
3�
20�
�
�������� 6�
14�
20�
�
� 23�
17�
40�
�
� �����������
� � ����������� 0.0005�
� Reset������������� 0.9999
�
� � ������������ 0.0010�
�
Applet available at : http://www.unc.edu/~preacher/fisher/fisher.htm
$�.�'��������� ��(�����Multivariate Tests(d)
Effect Value F
Hypothesis
df Error df Sig.
Partial Eta
Squared
Noncent.
Parameter
Intercept Pillai's Trace .963 439.425(b
) 5.000 84.000 .000 .963 2197.127
Wilks' Lambda .037 439.425(b
) 5.000 84.000 .000 .963 2197.127
Hotelling's Trace 26.156 439.425(b
) 5.000 84.000 .000 .963 2197.127
Roy's Largest Root
26.156 439.425(b) 5.000 84.000 .000 .963 2197.
127
group Pillai's Trace 1.317 8.540 20.00
0 348.000 .000 .329 170.804
Wilks' Lambda .126 12.117 20.00
0 279.54
6 .000 .404 189.538
Hotelling's Trace 3.824 15.775 20.00
0 330.000 .000 .489 315.506
Roy's Largest Root
2.957 51.454(c) 5.000 87.000 .000 .747 257.268
a Computed using alpha = .05 b Exact statistic c The statistic is an upper bound on F that yields a lower bound on the significance level. d Design: Intercept+group
Levene's Test of Equality of Error Variances(a)
F df1 df2 Sig. senBR 6.709 4 88 .000 senCC 4.797 4 88 .002 Rate 18.500 4 88 .000 CCdepth 12.360 4 88 .000 avgBRvolCyc 1.699 4 88 .157
Tests the null hypothesis that the error variance of the dependent variable is equal across groups. a Design: Intercept+group
Tests of Between-Subjects Effects
Source Dependent Variable
Type III Sum of
Squares df Mean Square F Sig. Partial Eta Squared
Corrected Model
senBR 47.489(b) 4 11.872 17.276 .000 .440
senCC 1346.311(c) 4 336.578 28.093 .000 .561 Rate 32734.677(d) 4 8183.669 24.000 .000 .522 CCdepth 4.141(e) 4 1.035 4.877 .001 .181 avgBRvolCyc 14863636.38
7(f) 4 3715909.097 30.854 .000 .584
Intercept senBR 393.900 1 393.900 573.189 .000 .867 senCC 14292.798 1 14292.798 1192.984 .000 .931 Rate 564126.667 1 564126.667 1654.403 .000 .949 CCdepth 226.232 1 226.232 1065.857 .000 .924 avgBRvolCyc 49997669.76
4 1 49997669.764 415.136 .000 .825
group senBR 47.489 4 11.872 17.276 .000 .440 senCC 1346.311 4 336.578 28.093 .000 .561 Rate 32734.677 4 8183.669 24.000 .000 .522 CCdepth 4.141 4 1.035 4.877 .001 .181 avgBRvolCyc 14863636.
387 4 3715909.097 30.854 .000 .584 Error senBR 60.474 8
8 .687 senCC 1054.303 8
8 11.981 Rate 30006.680 8
8 340.985 CCdepth 18.678 8
8 .212 avgBRvolCyc 10598441.46
3 88 120436.835
Total senBR 495.946 93
senCC 16656.798 93
Rate 630405.580 93
CCdepth 251.289 93
avgBRvolCyc 75422430.611
93
Corrected Total
senBR 107.964 92
senCC 2400.614 92
Rate 62741.357 92
CCdepth 22.819 92
avgBRvolCyc 25462077.850
92
a Computed using alpha = .05 b R Squared = .440 (Adjusted R Squared = .414) c R Squared = .561 (Adjusted R Squared = .541) d R Squared = .522 (Adjusted R Squared = .500) e R Squared = .181 (Adjusted R Squared = .144) f R Squared = .584 (Adjusted R Squared = .565)
Multiple Comparisons Dependent Variable
(I) group
(J) group
Mean Difference (I-
J) Std. Error Sig. 95% Confidence Interval
Lower Bound
Upper Bound
senBR Scheffe 1.00 2.00 -1.9997(*) .27675 .000 -2.8705 -1.1289 -1.1345(*) .26896 .003 -1.9808 -.2882 4.00 -1.8586(*) .27267 .000 -2.7166 -1.0007 5.00 -1.5174(*) .26557 .000 -2.3530 -.6817 2.00 1.00 1.9997(*) .27675 .000 1.1289 2.8705 3.00 .8651 .27675 .052 -.0057 1.7360 4.00 .1410 .28036 .992 -.7411 1.0232 5.00 .4823 .27347 .543 -.3782 1.3428 3.00 1.00 1.1345(*) .26896 .003 .2882 1.9808 2.00 -.8651 .27675 .052 -1.7360 .0057 4.00 -.7241 .27267 .143 -1.5821 .1339 5.00 -.3828 .26557 .722 -1.2185 .4528 4.00 1.00 1.8586(*) .27267 .000 1.0007 2.7166 2.00 -.1410 .28036 .992 -1.0232 .7411 3.00 .7241 .27267 .143 -.1339 1.5821 5.00 .3413 .26933 .807 -.5062 1.1888 5.00 1.00 1.5174(*) .26557 .000 .6817 2.3530 2.00 -.4823 .27347 .543 -1.3428 .3782 3.00 .3828 .26557 .722 -.4528 1.2185 4.00 -.3413 .26933 .807 -1.1888 .5062 Dunnett
T3 1.00 2.00 -1.9997(*) .35126 .000 -3.0479 -.9514
3.00 -1.1345(*) .31999 .013 -2.0967 -.1724 4.00 -1.8586(*) .30936 .000 -2.7949 -.9224 5.00 -1.5174(*) .29987 .000 -2.4311 -.6037 2.00 1.00 1.9997(*) .35126 .000 .9514 3.0479 3.00 .8651(*) .27638 .036 .0358 1.6945 4.00 .1410 .26399 1.000 -.6565 .9386 5.00 .4823 .25280 .468 -.2870 1.2517 3.00 1.00 1.1345(*) .31999 .013 .1724 2.0967 2.00 -.8651(*) .27638 .036 -1.6945 -.0358 4.00 -.7241(*) .22069 .023 -1.3810 -.0672 5.00 -.3828 .20718 .505 -1.0007 .2350 4.00 1.00 1.8586(*) .30936 .000 .9224 2.7949 2.00 -.1410 .26399 1.000 -.9386 .6565 3.00 .7241(*) .22069 .023 .0672 1.3810 5.00 .3413 .19034 .545 -.2255 .9081 5.00 1.00 1.5174(*) .29987 .000 .6037 2.4311 2.00 -.4823 .25280 .468 -1.2517 .2870 3.00 .3828 .20718 .505 -.2350 1.0007 4.00 -.3413 .19034 .545 -.9081 .2255 senCC Scheffe 1.00 2.00 -9.5041(*) 1.15556 .000 -13.1402 -5.8680 3.00 -8.2703(*) 1.12300 .000 -11.8039 -4.7366
4.00 -9.2843(*) 1.13849 .000 -12.8667 -5.7019 5.00 -10.1596(*) 1.10887 .000 -13.6488 -6.6705 2.00 1.00 9.5041(*) 1.15556 .000 5.8680 13.1402 3.00 1.2338 1.15556 .887 -2.4022 4.8699 4.00 .2198 1.17062 1.000 -3.4637 3.9033 5.00 -.6555 1.14183 .988 -4.2484 2.9374 3.00 1.00 8.2703(*) 1.12300 .000 4.7366 11.8039 2.00 -1.2338 1.15556 .887 -4.8699 2.4022 4.00 -1.0140 1.13849 .939 -4.5964 2.5683 5.00 -1.8894 1.10887 .577 -5.3786 1.5998 4.00 1.00 9.2843(*) 1.13849 .000 5.7019 12.8667 2.00 -.2198 1.17062 1.000 -3.9033 3.4637 3.00 1.0140 1.13849 .939 -2.5683 4.5964 5.00 -.8753 1.12456 .962 -4.4139 2.6632 5.00 1.00 10.1596(*) 1.10887 .000 6.6705 13.6488 2.00 .6555 1.14183 .988 -2.9374 4.2484 3.00 1.8894 1.10887 .577 -1.5998 5.3786 4.00 .8753 1.12456 .962 -2.6632 4.4139 Dunnett
T3 1.00 2.00 -9.5041(*) 1.56385 .000 -14.1679 -4.8403
3.00 -8.2703(*) 1.20524 .000 -11.9255 -4.6151 4.00 -9.2843(*) 1.21374 .000 -12.9608 -5.6078 5.00 -10.1596(*) 1.20531 .000 -13.8141 -6.5052 2.00 1.00 9.5041(*) 1.56385 .000 4.8403 14.1679 3.00 1.2338 1.25689 .974 -2.6248 5.0925 4.00 .2198 1.26504 1.000 -3.6580 4.0976 5.00 -.6555 1.25696 1.000 -4.5135 3.2025 3.00 1.00 8.2703(*) 1.20524 .000 4.6151 11.9255 2.00 -1.2338 1.25689 .974 -5.0925 2.6248 4.00 -1.0140 .77930 .874 -3.3327 1.3046 5.00 -1.8894 .76611 .162 -4.1609 .3821 4.00 1.00 9.2843(*) 1.21374 .000 5.6078 12.9608 2.00 -.2198 1.26504 1.000 -4.0976 3.6580 3.00 1.0140 .77930 .874 -1.3046 3.3327 5.00 -.8753 .77941 .944 -3.1909 1.4403 5.00 1.00 10.1596(*) 1.20531 .000 6.5052 13.8141 2.00 .6555 1.25696 1.000 -3.2025 4.5135 3.00 1.8894 .76611 .162 -.3821 4.1609 4.00 .8753 .77941 .944 -1.4403 3.1909 Rate Scheffe 1.00 2.00 -32.2662(*) 6.16478 .000 -51.6643 -12.8681 3.00 -52.1210(*) 5.99109 .000 -70.9725 -33.2694 4.00 -40.6254(*) 6.07373 .000 -59.7370 -21.5137 5.00 -47.6579(*) 5.91573 .000 -66.2724 -29.0435 2.00 1.00 32.2662(*) 6.16478 .000 12.8681 51.6643 3.00 -19.8547(*) 6.16478 .042 -39.2529 -.4566 4.00 -8.3592 6.24512 .774 -28.0101 11.2918 5.00 -15.3917 6.09157 .182 -34.5594 3.7760 3.00 1.00 52.1210(*) 5.99109 .000 33.2694 70.9725 2.00 19.8547(*) 6.16478 .042 .4566 39.2529 4.00 11.4956 6.07373 .470 -7.6160 30.6072
5.00 4.4630 5.91573 .966 -14.1514 23.0775 4.00 1.00 40.6254(*) 6.07373 .000 21.5137 59.7370 2.00 8.3592 6.24512 .774 -11.2918 28.0101 3.00 -11.4956 6.07373 .470 -30.6072 7.6160 5.00 -7.0326 5.99941 .848 -25.9103 11.8452 5.00 1.00 47.6579(*) 5.91573 .000 29.0435 66.2724 2.00 15.3917 6.09157 .182 -3.7760 34.5594 3.00 -4.4630 5.91573 .966 -23.0775 14.1514 4.00 7.0326 5.99941 .848 -11.8452 25.9103 Dunnett
T3 1.00 2.00 -32.2662(*) 8.87102 .009 -58.7902 -5.7422
3.00 -52.1210(*) 7.51938 .000 -75.3041 -28.9378 4.00 -40.6254(*) 7.68727 .000 -64.1647 -17.0860 5.00 -47.6579(*) 7.21941 .000 -70.2703 -25.0455 2.00 1.00 32.2662(*) 8.87102 .009 5.7422 58.7902 3.00 -19.8547(*) 5.80733 .023 -37.7281 -1.9814 4.00 -8.3592 6.02313 .824 -26.7250 10.0067 5.00 -15.3917 5.41330 .095 -32.5007 1.7173 3.00 1.00 52.1210(*) 7.51938 .000 28.9378 75.3041 2.00 19.8547(*) 5.80733 .023 1.9814 37.7281 4.00 11.4956(*) 3.75823 .041 .2865 22.7047 5.00 4.4630 2.67396 .638 -3.6576 12.5837 4.00 1.00 40.6254(*) 7.68727 .000 17.0860 64.1647 2.00 8.3592 6.02313 .824 -10.0067 26.7250 3.00 -11.4956(*) 3.75823 .041 -22.7047 -.2865 5.00 -7.0326 3.11497 .268 -16.6186 2.5535 5.00 1.00 47.6579(*) 7.21941 .000 25.0455 70.2703 2.00 15.3917 5.41330 .095 -1.7173 32.5007 3.00 -4.4630 2.67396 .638 -12.5837 3.6576 4.00 7.0326 3.11497 .268 -2.5535 16.6186 CCdepth Scheffe 1.00 2.00 -.1413 .15381 .932 -.6253 .3427 3.00 -.5179(*) .14947 .023 -.9883 -.0476 4.00 -.3751 .15154 .200 -.8520 .1017 5.00 -.5266(*) .14759 .017 -.9911 -.0622 2.00 1.00 .1413 .15381 .932 -.3427 .6253 3.00 -.3766 .15381 .209 -.8606 .1073 4.00 -.2338 .15581 .690 -.7241 .2565 5.00 -.3853 .15198 .180 -.8635 .0929 3.00 1.00 .5179(*) .14947 .023 .0476 .9883 2.00 .3766 .15381 .209 -.1073 .8606 4.00 .1428 .15154 .925 -.3340 .6196 5.00 -.0087 .14759 1.000 -.4731 .4557 4.00 1.00 .3751 .15154 .200 -.1017 .8520 2.00 .2338 .15581 .690 -.2565 .7241 3.00 -.1428 .15154 .925 -.6196 .3340 5.00 -.1515 .14968 .905 -.6225 .3195 5.00 1.00 .5266(*) .14759 .017 .0622 .9911 2.00 .3853 .15198 .180 -.0929 .8635 3.00 .0087 .14759 1.000 -.4557 .4731 4.00 .1515 .14968 .905 -.3195 .6225
Dunnett T3
1.00 2.00 -.1413 .20553 .998 -.7737 .4911
3.00 -.5179 .20622 .166 -1.1516 .1158 4.00 -.3751 .20716 .538 -1.0109 .2606 5.00 -.5266 .20105 .138 -1.1499 .0966 2.00 1.00 .1413 .20553 .998 -.4911 .7737 3.00 -.3766(*) .09965 .006 -.6736 -.0797 4.00 -.2338 .10159 .232 -.5372 .0695 5.00 -.3853(*) .08845 .001 -.6501 -.1206 3.00 1.00 .5179 .20622 .166 -.1158 1.1516 2.00 .3766(*) .09965 .006 .0797 .6736 4.00 .1428 .10298 .828 -.1636 .4492 5.00 -.0087 .09004 1.000 -.2769 .2595 4.00 1.00 .3751 .20716 .538 -.2606 1.0109 2.00 .2338 .10159 .232 -.0695 .5372 3.00 -.1428 .10298 .828 -.4492 .1636 5.00 -.1515 .09219 .655 -.4273 .1243 5.00 1.00 .5266 .20105 .138 -.0966 1.1499 2.00 .3853(*) .08845 .001 .1206 .6501 3.00 .0087 .09004 1.000 -.2595 .2769 4.00 .1515 .09219 .655 -.1243 .4273 avgBRvolCyc
Scheffe 1.00 2.00 -857.9926(*) 115.85897 .000 -
1222.5548 -493.4304
3.00 -335.8544 112.59468 .073 -690.1451 18.4364
4.00 -742.9189(*) 114.14778 .000 -
1102.0967 -383.7412
5.00 -1110.1317(*) 111.17834 .000 -
1459.9658 -760.2976
2.00 1.00 857.9926(*) 115.85897 .000 493.4304 1222.5548
3.00 522.1383(*) 115.85897 .001 157.5761 886.7004
4.00 115.0737 117.36888 .915 -254.2396 484.3869
5.00 -252.1391 114.48302 .311 -612.3717 108.0935
3.00 1.00 335.8544 112.59468 .073 -18.4364 690.1451
2.00 -522.1383(*) 115.85897 .001 -886.7004 -157.5761
4.00 -407.0646(*) 114.14778 .017 -766.2423 -47.8869
5.00 -774.2774(*) 111.17834 .000 -
1124.1114 -424.4433
4.00 1.00 742.9189(*) 114.14778 .000 383.7412 1102.0967
2.00 -115.0737 117.36888 .915 -484.3869 254.2396
3.00 407.0646(*) 114.14778 .017 47.8869 766.2423
5.00 -367.2128(*) 112.75095 .038 -721.9952 -12.4303
5.00 1.00 1110.1317(*) 111.17834 .000 760.2976 1459.9658
2.00 252.1391 114.48302 .311 -108.0935 612.3717
3.00 774.2774(*) 111.17834 .000 424.4433 1124.1114
4.00 367.2128(*) 112.75095 .038 12.4303 721.9952
Dunnett T3
1.00 2.00 -857.9926(*) 104.04305 .000 -
1176.5306 -539.4546
3.00 -335.8544(*) 86.30985 .005 -594.7625 -76.9462
4.00 -742.9189(*) 98.37015 .000 -
1041.5325 -444.3054
5.00 -1110.1317(*) 105.62110 .000 -
1429.4408 -790.8226
2.00 1.00 857.9926(*) 104.04305 .000 539.4546 1176.5306
3.00 522.1383(*) 117.91064 .001 168.7549 875.5216
4.00 115.0737 127.00478 .986 -264.3643 494.5117
5.00 -252.1391 132.70020 .467 -646.9770 142.6988
3.00 1.00 335.8544(*) 86.30985 .005 76.9462 594.7625
2.00 -522.1383(*) 117.91064 .001 -875.5216 -168.7549
4.00 -407.0646(*) 112.93648 .010 -743.8516 -70.2775
5.00 -774.2774(*) 119.30540 .000 -
1129.1087 -419.4461
4.00 1.00 742.9189(*) 98.37015 .000 444.3054 1041.5325
2.00 -115.0737 127.00478 .986 -494.5117 264.3643
3.00 407.0646(*) 112.93648 .010 70.2775 743.8516
5.00 -367.2128 128.30072 .065 -748.2371 13.8115
5.00 1.00 1110.1317(*) 105.62110 .000 790.8226 1429.4408
2.00 252.1391 132.70020 .467 -142.6988 646.9770
3.00 774.2774(*) 119.30540 .000 419.4461 1129.1087
4.00 367.2128 128.30072 .065 -13.8115 748.2371
Based on observed means. * The mean difference is significant at the .05 level.
����/���������������� Cohen’s Kappa calculation”
1900
3521
0223
Scram
Scram
Opp
TactOpp
Tact
Rater 2
Rat
er 1
24
19
56
25
2254 100
Expected Frequencies
ef =Row total * column total
Overall total
ef1 = 6
ef2 = 30.4
ef3 = 4.18
SumDiag = 96
SumEF = 40.6
Kappa Calculation
96 – 40.6
κ =N - SumEF
SumDiag - SumEF
κ =100 – 40.6
κ > 0.70 considered acceptable
= 0.899
MANOVA for COCOM coding Descriptive Statistics
group Mean Std. Deviation N 1.00 .8985 .12223 20 2.00 .2980 .14753 20 3.00 .1870 .22679 20 4.00 .1710 .11805 20 5.00 .1580 .09373 20
perSCRAM
Total .3425 .31911 100 1.00 .0455 .06886 20 2.00 .6595 .15357 20 3.00 .1660 .07783 20 4.00 .6190 .16370 20 5.00 .6010 .09580 20
perOPP
Total .4182 .28473 100 1.00 .0555 .07776 20 2.00 .0425 .08491 20 3.00 .6450 .25078 20 4.00 .2110 .17976 20 5.00 .2400 .12053 20
perTACT
Total .2388 .26770 100 Multivariate Tests(c)
Effect Value F Hypothesis df Error df Sig. Pillai's Trace 1.000 333353.53
3(a) 3.000 93.000 .000
Wilks' Lambda .000 333353.533(a) 3.000 93.000 .000
Hotelling's Trace 10753.340 333353.53
3(a) 3.000 93.000 .000
Intercept
Roy's Largest Root 10753.340 333353.53
3(a) 3.000 93.000 .000
Pillai's Trace 1.525 24.561 12.000 285.000 .000 Wilks' Lambda .049 43.600 12.000 246.346 .000
Hotelling's Trace 7.718 58.958 12.000 275.000 .000
group
Roy's Largest Root 5.678 134.842(b) 4.000 95.000 .000
a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+group
Tests of Between-Subjects Effects
Source Dependent Variable
Type III Sum of Squares df
Mean Square F Sig.
Corrected Model perSCRAM 7.975(a) 4 1.994 89.924 .000 perOPP 6.689(b) 4 1.672 118.844 .000 perTACT 4.758(c) 4 1.190 48.360 .000 Intercept perSCRAM 11.731 1 11.731 529.085 .000 perOPP 17.489 1 17.489 1242.841 .000 perTACT 5.703 1 5.703 231.836 .000 group perSCRAM 7.975 4 1.994 89.924 .000 perOPP 6.689 4 1.672 118.844 .000 perTACT 4.758 4 1.190 48.360 .000 Error perSCRAM 2.106 95 .022 perOPP 1.337 95 .014 perTACT 2.337 95 .025 Total perSCRAM 21.812 100 perOPP 25.515 100 perTACT 12.797 100 Corrected Total perSCRAM 10.081 99 perOPP 8.026 99 perTACT 7.095 99
a R Squared = .791 (Adjusted R Squared = .782) b R Squared = .833 (Adjusted R Squared = .826) c R Squared = .671 (Adjusted R Squared = .657) Multiple Comparisons Scheffe
Dependent Variable (I) group (J) group
Mean Difference
(I-J) Std. Error Sig. 95% Confidence
Interval
Lower Bound
Upper
Bound
perSCRAM 1.00 2.00 .6005(*) .04709 .000 .4526 .7484 3.00 .7115(*) .04709 .000 .5636 .8594 4.00 .7275(*) .04709 .000 .5796 .8754 5.00 .7405(*) .04709 .000 .5926 .8884 2.00 1.00 -.6005(*) .04709 .000 -.7484 -
.4526 3.00 .1110 .04709 .244 -.0369 .2589 4.00 .1270 .04709 .132 -.0209 .2749 5.00 .1400 .04709 .074 -.0079 .2879 3.00 1.00 -.7115(*) .04709 .000 -.8594 -
.5636 2.00 -.1110 .04709 .244 -.2589 .0369 4.00 .0160 .04709 .998 -.1319 .1639 5.00 .0290 .04709 .984 -.1189 .1769 4.00 1.00 -.7275(*) .04709 .000 -.8754 -
.5796
2.00 -.1270 .04709 .132 -.2749 .0209 3.00 -.0160 .04709 .998 -.1639 .1319 5.00 .0130 .04709 .999 -.1349 .1609 5.00 1.00 -.7405(*) .04709 .000 -.8884 -
.5926 2.00 -.1400 .04709 .074 -.2879 .0079 3.00 -.0290 .04709 .984 -.1769 .1189 4.00 -.0130 .04709 .999 -.1609 .1349 perOPP 1.00 2.00 -.6140(*) .03751 .000 -.7319 -
.4961 3.00 -.1205(*) .03751 .042 -.2384 -
.0026 4.00 -.5735(*) .03751 .000 -.6914 -
.4556 5.00 -.5555(*) .03751 .000 -.6734 -
.4376 2.00 1.00 .6140(*) .03751 .000 .4961 .7319 3.00 .4935(*) .03751 .000 .3756 .6114 4.00 .0405 .03751 .883 -.0774 .1584 5.00 .0585 .03751 .658 -.0594 .1764 3.00 1.00 .1205(*) .03751 .042 .0026 .2384 2.00 -.4935(*) .03751 .000 -.6114 -
.3756 4.00 -.4530(*) .03751 .000 -.5709 -
.3351 5.00 -.4350(*) .03751 .000 -.5529 -
.3171 4.00 1.00 .5735(*) .03751 .000 .4556 .6914 2.00 -.0405 .03751 .883 -.1584 .0774 3.00 .4530(*) .03751 .000 .3351 .5709 5.00 .0180 .03751 .994 -.0999 .1359 5.00 1.00 .5555(*) .03751 .000 .4376 .6734 2.00 -.0585 .03751 .658 -.1764 .0594 3.00 .4350(*) .03751 .000 .3171 .5529 4.00 -.0180 .03751 .994 -.1359 .0999 perTACT 1.00 2.00 .0130 .04960 .999 -.1428 .1688 3.00 -.5895(*) .04960 .000 -.7453 -
.4337 4.00 -.1555 .04960 .051 -.3113 .0003 5.00 -.1845(*) .04960 .011 -.3403 -
.0287 2.00 1.00 -.0130 .04960 .999 -.1688 .1428 3.00 -.6025(*) .04960 .000 -.7583 -
.4467 4.00 -.1685(*) .04960 .026 -.3243 -
.0127 5.00 -.1975(*) .04960 .005 -.3533 -
.0417 3.00 1.00 .5895(*) .04960 .000 .4337 .7453 2.00 .6025(*) .04960 .000 .4467 .7583 4.00 .4340(*) .04960 .000 .2782 .5898 5.00 .4050(*) .04960 .000 .2492 .5608 4.00 1.00 .1555 .04960 .051 -.0003 .3113 2.00 .1685(*) .04960 .026 .0127 .3243
3.00 -.4340(*) .04960 .000 -.5898 -.2782
5.00 -.0290 .04960 .987 -.1848 .1268 5.00 1.00 .1845(*) .04960 .011 .0287 .3403 2.00 .1975(*) .04960 .005 .0417 .3533 3.00 -.4050(*) .04960 .000 -.5608 -
.2492 4.00 .0290 .04960 .987 -.1268 .1848
Based on observed means. * The mean difference is significant at the .05 level.
$�.�'���������$������� Descriptive Statistics
group Mean Std. Deviation N perSCRAM 1.00 .8985 .12223 20 2.00 .2980 .14753 20 3.00 .1870 .22679 20 4.00 .1710 .11805 20 5.00 .1580 .09373 20 Total .3425 .31911 100 perOPP 1.00 .0455 .06886 20 2.00 .6595 .15357 20 3.00 .1660 .07783 20 4.00 .6190 .16370 20 5.00 .6010 .09580 20 Total .4182 .28473 100 perTACT 1.00 .0555 .07776 20 2.00 .0425 .08491 20 3.00 .6450 .25078 20 4.00 .2110 .17976 20 5.00 .2400 .12053 20 Total .2388 .26770 100
Multivariate Tests(c)
Effect Value F Hypothesis df Error df Sig. Pillai's Trace 1.000 333353.53
3(a) 3.000 93.000 .000
Wilks' Lambda .000 333353.533(a) 3.000 93.000 .000
Hotelling's Trace 10753.340 333353.53
3(a) 3.000 93.000 .000
Intercept
Roy's Largest Root 10753.340 333353.53
3(a) 3.000 93.000 .000
Pillai's Trace 1.525 24.561 12.000 285.000 .000 Wilks' Lambda .049 43.600 12.000 246.346 .000 Hotelling's Trace 7.718 58.958 12.000 275.000 .000
group
Roy's Largest Root 5.678 134.842(b) 4.000 95.000 .000
a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+group
Tests of Between-Subjects Effects
Source Dependent Variable
Type III Sum of
Squares df Mean Square F Sig. Corrected Model perSCRAM 7.975(a) 4 1.994 89.924 .000 perOPP 6.689(b) 4 1.672 118.844 .000 perTACT 4.758(c) 4 1.190 48.360 .000 Intercept perSCRAM 11.731 1 11.731 529.085 .000 perOPP 17.489 1 17.489 1242.841 .000 perTACT 5.703 1 5.703 231.836 .000 group perSCRAM 7.975 4 1.994 89.924 .000 perOPP 6.689 4 1.672 118.844 .000 perTACT 4.758 4 1.190 48.360 .000 Error perSCRAM 2.106 95 .022 perOPP 1.337 95 .014 perTACT 2.337 95 .025 Total perSCRAM 21.812 100 perOPP 25.515 100 perTACT 12.797 100 Corrected Total perSCRAM 10.081 99 perOPP 8.026 99 perTACT 7.095 99
a R Squared = .791 (Adjusted R Squared = .782) b R Squared = .833 (Adjusted R Squared = .826) c R Squared = .671 (Adjusted R Squared = .657) Multiple Comparisons Scheffe
Dependent Variable (I) group
(J) group
Mean Difference
(I-J) Std. Error Sig. 95% Confidence Interval
Lower Bound
Upper Bound
perSCRAM 1.00 2.00 .6005(*) .04709 .000 .4526 .7484 3.00 .7115(*) .04709 .000 .5636 .8594 4.00 .7275(*) .04709 .000 .5796 .8754 5.00 .7405(*) .04709 .000 .5926 .8884 2.00 1.00 -.6005(*) .04709 .000 -.7484 -.4526 3.00 .1110 .04709 .244 -.0369 .2589 4.00 .1270 .04709 .132 -.0209 .2749 5.00 .1400 .04709 .074 -.0079 .2879 3.00 1.00 -.7115(*) .04709 .000 -.8594 -.5636 2.00 -.1110 .04709 .244 -.2589 .0369 4.00 .0160 .04709 .998 -.1319 .1639 5.00 .0290 .04709 .984 -.1189 .1769 4.00 1.00 -.7275(*) .04709 .000 -.8754 -.5796 2.00 -.1270 .04709 .132 -.2749 .0209 3.00 -.0160 .04709 .998 -.1639 .1319 5.00 .0130 .04709 .999 -.1349 .1609
5.00 1.00 -.7405(*) .04709 .000 -.8884 -.5926 2.00 -.1400 .04709 .074 -.2879 .0079 3.00 -.0290 .04709 .984 -.1769 .1189 4.00 -.0130 .04709 .999 -.1609 .1349 perOPP 1.00 2.00 -.6140(*) .03751 .000 -.7319 -.4961 3.00 -.1205(*) .03751 .042 -.2384 -.0026 4.00 -.5735(*) .03751 .000 -.6914 -.4556 5.00 -.5555(*) .03751 .000 -.6734 -.4376 2.00 1.00 .6140(*) .03751 .000 .4961 .7319 3.00 .4935(*) .03751 .000 .3756 .6114 4.00 .0405 .03751 .883 -.0774 .1584 5.00 .0585 .03751 .658 -.0594 .1764 3.00 1.00 .1205(*) .03751 .042 .0026 .2384 2.00 -.4935(*) .03751 .000 -.6114 -.3756 4.00 -.4530(*) .03751 .000 -.5709 -.3351 5.00 -.4350(*) .03751 .000 -.5529 -.3171 4.00 1.00 .5735(*) .03751 .000 .4556 .6914 2.00 -.0405 .03751 .883 -.1584 .0774 3.00 .4530(*) .03751 .000 .3351 .5709 5.00 .0180 .03751 .994 -.0999 .1359 5.00 1.00 .5555(*) .03751 .000 .4376 .6734 2.00 -.0585 .03751 .658 -.1764 .0594 3.00 .4350(*) .03751 .000 .3171 .5529 4.00 -.0180 .03751 .994 -.1359 .0999 perTACT 1.00 2.00 .0130 .04960 .999 -.1428 .1688 3.00 -.5895(*) .04960 .000 -.7453 -.4337 4.00 -.1555 .04960 .051 -.3113 .0003 5.00 -.1845(*) .04960 .011 -.3403 -.0287 2.00 1.00 -.0130 .04960 .999 -.1688 .1428 3.00 -.6025(*) .04960 .000 -.7583 -.4467 4.00 -.1685(*) .04960 .026 -.3243 -.0127 5.00 -.1975(*) .04960 .005 -.3533 -.0417 3.00 1.00 .5895(*) .04960 .000 .4337 .7453 2.00 .6025(*) .04960 .000 .4467 .7583 4.00 .4340(*) .04960 .000 .2782 .5898 5.00 .4050(*) .04960 .000 .2492 .5608 4.00 1.00 .1555 .04960 .051 -.0003 .3113 2.00 .1685(*) .04960 .026 .0127 .3243 3.00 -.4340(*) .04960 .000 -.5898 -.2782 5.00 -.0290 .04960 .987 -.1848 .1268 5.00 1.00 .1845(*) .04960 .011 .0287 .3403 2.00 .1975(*) .04960 .005 .0417 .3533 3.00 -.4050(*) .04960 .000 -.5608 -.2492 4.00 .0290 .04960 .987 -.1268 .1848
Based on observed means. * The mean difference is significant at the .05 level.
$�.�'�����!� �/����0�� ����������������� ��� � Descriptive Statistics
grpnum Mean Std. Deviation N 1.00 1.6000 1.46539 20 2.00 7.8000 1.05631 20 3.00 7.8500 1.49649 20 4.00 8.4500 .88704 20 5.00 8.6500 .48936 20
Yprotocol
Total 6.8700 2.89428 100 1.00 6.0500 1.66938 20 2.00 5.7500 1.29269 20 3.00 8.2500 1.11803 20 4.00 6.7500 1.74341 20 5.00 7.8500 1.08942 20
action
Total 6.9300 1.69524 100 1.00 2.8500 2.15883 20 2.00 7.8000 1.85245 20 3.00 2.4500 2.39462 20 4.00 7.0000 1.89181 20 5.00 7.6000 2.21003 20
FB
Total 5.5400 3.16042 100 1.00 6.5500 1.90498 20 2.00 6.0500 .99868 20 3.00 6.4000 2.01050 20 4.00 5.6500 1.13671 20 5.00 6.6500 1.84320 20
goal
Total 6.2600 1.64298 100 1.00 7.0000 2.65568 20 2.00 6.8000 2.28496 20 3.00 7.5500 2.01246 20 4.00 7.7500 2.22131 20 5.00 7.8000 2.52566 20
Tpressure
Total 7.3800 2.33887 100 1.00 4.7500 2.12442 20 2.00 4.6500 1.63111 20 3.00 3.8000 1.00525 20 4.00 4.8000 1.57614 20 5.00 3.8000 .76777 20
time
Total 4.3600 1.54082 100 1.00 3.8500 2.34577 20 2.00 7.1500 1.95408 20 3.00 5.6000 2.18608 20 4.00 7.3000 2.55672 20
outcome
5.00 7.3000 2.17885 20
Total 6.2400 2.59417 100 1.00 4.6500 2.05900 20 2.00 1.7500 3.00657 20 3.00 2.2000 1.96281 20 4.00 1.2000 1.90843 20 5.00 2.8500 2.87045 20
words
Total 2.5300 2.64558 100 Multivariate Tests(c)
Effect Value F Hypothesis df Error df Sig. Pillai's Trace .989 985.561(a) 8.000 88.000 .000 Wilks' Lambda .011 985.561(a) 8.000 88.000 .000 Hotelling's Trace 89.596 985.561(a) 8.000 88.000 .000
Intercept
Roy's Largest Root 89.596 985.561(a) 8.000 88.000 .000
Pillai's Trace 1.618 7.730 32.000 364.000 .000 Wilks' Lambda .047 13.191 32.000 326.123 .000 Hotelling's Trace 8.363 22.607 32.000 346.000 .000
grpnum
Roy's Largest Root 6.991 79.518(b) 8.000 91.000 .000
a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+grpnum Tests of Between-Subjects Effects
Source Dependent Variable
Type III Sum of Squares df Mean Square F Sig.
Corrected Model
Yprotocol 705.260(a) 4 176.315 135.026 .000
action 95.760(b) 4 23.940 12.049 .000 FB 565.340(c) 4 141.335 31.704 .000 goal 13.440(d) 4 3.360 1.258 .292 Tpressure 16.460(e) 4 4.115 .744 .564 time 21.140(f) 4 5.285 2.347 .060 outcome 183.940(g) 4 45.985 9.058 .000 words 141.660(h) 4 35.415 6.103 .000 Intercept Yprotocol 4719.690 1 4719.690 3614.434 .000 action 4802.490 1 4802.490 2417.147 .000 FB 3069.160 1 3069.160 688.477 .000 goal 3918.760 1 3918.760 1466.833 .000 Tpressure 5446.440 1 5446.440 985.359 .000 time 1900.960 1 1900.960 844.279 .000 outcome 3893.760 1 3893.760 766.965 .000 words 640.090 1 640.090 110.310 .000 grpnum Yprotocol 705.260 4 176.315 135.026 .000
action 95.760 4 23.940 12.049 .000 FB 565.340 4 141.335 31.704 .000 goal 13.440 4 3.360 1.258 .292 Tpressure 16.460 4 4.115 .744 .564 time 21.140 4 5.285 2.347 .060 outcome 183.940 4 45.985 9.058 .000 words 141.660 4 35.415 6.103 .000 Error Yprotocol 124.050 95 1.306 action 188.750 95 1.987 FB 423.500 95 4.458 goal 253.800 95 2.672 Tpressure 525.100 95 5.527 time 213.900 95 2.252 outcome 482.300 95 5.077 words 551.250 95 5.803 Total Yprotocol 5549.000 100 action 5087.000 100 FB 4058.000 100 goal 4186.000 100 Tpressure 5988.000 100 time 2136.000 100 outcome 4560.000 100 words 1333.000 100 Corrected Total
Yprotocol 829.310 99
action 284.510 99 FB 988.840 99 goal 267.240 99 Tpressure 541.560 99 time 235.040 99 outcome 666.240 99 words 692.910 99
a R Squared = .850 (Adjusted R Squared = .844) b R Squared = .337 (Adjusted R Squared = .309) c R Squared = .572 (Adjusted R Squared = .554) d R Squared = .050 (Adjusted R Squared = .010) e R Squared = .030 (Adjusted R Squared = -.010) f R Squared = .090 (Adjusted R Squared = .052) g R Squared = .276 (Adjusted R Squared = .246) h R Squared = .204 (Adjusted R Squared = .171)
% "�* ��#������ �+���� ����� ����%������&��Class 1 represents CPR-DEV while class 2 = GRE-NO.
Gp1 is the number of subjects that received feedback (2 or more cases) during the scneario.
Gp 2 is the number of participants that are considered to not have received feedback.
The one-tail p value of 0.0267 suggests a significant difference.
��������� ��� ��� �
�������� 8�
12�
20�
�
�������� 15�
5�
20�
�
� 23�
17�
40�
�
� �����������
� � ����������� 0.0267�
� Reset������������� 0.9952
�
� � ������������ 0.0535�
��������� Status okay�
�
Applet available at : http://www.unc.edu/~preacher/fisher/fisher.htm
% "�* ��#������ �+��!���������1������2�
Class 1 represents all-DEV while class 2 = all-NODEV.
Gp1 is the number of subjects that performed the proper sequence.
Gp 2 is the number of participants failing to follow proper sequence.
��������� ��� ��� �
�������� 45�
15�
60�
�
�������� 15�
32�
47�
�
� 60�
47�
107�
�
� �����������
� � ����������� 0.0000�
� Reset������������� 0.9999
�
� � ������������ 0.0000�
��������� Status okay�
�
Class 1 represents all-DEV while class 2 = CPR-NO.
Gp1 is the number of subjects that performed the proper sequence.
Gp 2 is the number of participants failing to follow proper sequence.
��������� ��� ��� �
�������� 45�
8�
53�
�
�������� 15�
12�
27�
�
� 60�
20�
80�
�
� �����������
� � ����������� 0.0053�
� Reset������������� 0.9989
�
� � ������������ 0.0063�
��������� Status okay�
�
Both tests are significant
)������.��!�"# � ������1"������)������.��!�"# � ������1"������)������.��!�"# � ������1"������)������.��!�"# � ������1"����������
THIS TRAINING DOES NOT RESULT IN CPR CERTIFICATION
Thank you for volunteering to participate in our study. Today you will be taught a modified version of cardiopulmonary resuscitation or CPR. While what you are being taught today would be useful, if you are ever in a situation where CPR is needed, this training in no way makes you certified in CPR. We have changed steps and added equipment that alter this training from a certified course. The training will consist of a lecture time where the steps of CPR are demonstrated and explained. After this, all participates will be split up into 2 groups. While you are in your group you will be asked to “administer” CPR to a victim. You will then follow the steps of resuscitation you were taught in the demonstration. While in your group, if you are not the rescuer you will be asked to follow the steps on the handout to make sure all are followed. Name _______________________________________ (print) Signature____________________________________ Date_____________
)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/������
Courtesy of the American Heart Association, available at: http://circ.ahajournals.org/content/vol102/issue90001/images/large/hc33t0071002.jpeg
)������.��!������'����������)������.��!������'����������)������.��!������'����������)������.��!������'�������������������� �������������� ��� �������������� ��� �������������� ��� �������������� �����The value of a JITS system can be realized on multiple dimensions. It may
empower an intermediate user to perform at an expert level. It may enable a novice to
accomplish a formerly unachievable task. It may even support an expert in a degraded
state or help prevent errors.
Experts are not immune to the perturbations promulgated through innovation.
Manufacturing and the healthcare industry are two exemplar domains that have seen
significant technological infusion over the last fifty years. Rarely can technology be
introduced without notably impacting the human element of the system (Sheridan, 2002).
Requirements, procedures, and resources are often significantly altered as a result of new
technology. It is no surprise that knowledge requirements, skills, and attitudes are
impacted as well.
In emergency response, trained, experienced responders could also benefit
from JITS systems. The problems of CPR effectiveness and protocol adherence extend
beyond novice performers. Section 1 described the problems of improper ventilations,
inadequate compression rates and depths, and numerous accounts of rib and sternum
fractures (Abella et al., 2005; Lederer, Mair, Rabl, & Baubin, 2004; Myklebust, et al.,
2005; Wik, et al., 2005).
Abella et al. (2005) assert attaining “high-quality CPR” requires a method to
“improve monitoring and feedback” (p. 309). This is exactly the expert use paradigm of
JITS. Ultimately, the appropriate feedback could allow professionals to customize CPR
based on the individual victim’s needs (such as altering the protocol for larger breaths,
more compressions, etc.). Sensors, providing oxygen saturation or end-tidal CO2, could
provide experts valuable feedback during the procedure and augment their response.
Experts have invested considerable time and effort forging their proficiency.
Becoming expert in a field is so demanding that their breadth of expertise is usually
limited (Lukasiewicz, 1994, Moghaddam, 1997). Thus, they are not likely to perform as
experts when novel tasks and new technologies are thrust upon them. They will not have
automatized the relevant skills, increasing the drain on their cognitive resources. Experts
may be stripped of their status (at least temporarily) and find themselves acting as non-
experts. JITS solutions could serve them well.
CPR & Defibrillation A man collapses in a mall and is in need of immediate medical attention. He has
no pulse, and is not breathing. A woman, with no medical training, seizes a device that
contains supplies and instructions for administering basic life support (BLS). She opens
the box and a video screen engages her attention and provides step-by-step instructions
on how to complete each task. She removes the victim’s shirt and places electrode pads
correctly because both steps are demonstrated with simple dynamic graphics mapping the
salient features on the display to the real-world. The pads not only serve to monitor and
treat the patient, but also collect data on the actions of the user. This function enables the
system to provide corrective feedback (the frequency of chest compressions for example).
Similar monitoring technology resides in the oxygen mask. Thoughtful engineering and
animated instructions ensure proper placement on the victim. The system prescribes an
action sequence based on the needs of the patient. A protocol is devised, communicated
with cues and feedback, enabling the naïve caregiver to administer life-saving support.
Help is on the way as a call to 911 was placed upon activating the support system.
Fire extinguisher
Some oily rags are ignited in a homeowner’s garage. He runs to get his fire
extinguisher and dons the accompanying goggles. These are not high school chemistry
goggles (though eye protection is a key feature). They contain a microprocessor, infra-
red sensors, and an ability to overlay images on the real world. The images are
components of a larger set of instructions presented to the user. The Spartan pull, aim,
squeeze, sweep directions are conveyed to the user in the proper sequence just as he needs
the information. Sensors on the extinguisher itself support the tutorial. For example, the
spray & sweep commands are not activated until pin removal is registered. Guidance is
given when the user peers through the goggles at the fire. A synthetic vision program
provides a moving target to optimize the spray & sweep action. The sensors in the
goggles also calculate the distance of the operator from the fire and whether he should
move closer or back up. By comparing the intensity of the fire with the capacity of the
extinguisher (and changing intensity as a result of the extinguishing agent applied), a
decision algorithm continually runs to determine if the fire should be battled or if/when to
evacuate. Through dynamic temperature readings, the system determines the user is
aiming too high and provides instructive feedback to correct the error. Now the fuel is
being suffocated and quenched and the fire soon dies.
Space station maintenance
An astronaut is preparing a spacewalk to repair a component on the International
Space Station (ISS). Preparation for this task began several years ago. She was first
introduced to this task on a desktop simulator. A JITS system guided her through her
first actions and began to sketch her profile from the first encounter. Subsequent
practice with the system not only educated the astronaut about the task, but the system
learned a great deal about the user. The JITS system was integrated in every step of
training and practice from part task simulations to the pool. This system is also utilized
when performing the actual spacewalks. The user and system have developed a
significant history over the years of training. The information requirements and data
presentation have adjusted with the astronaut’s evolving competency. The system has a
robust representation of the user for each task based on parameters such as the number of
interactions, ratio of successful interactions, and even knows how long she takes to
perform individual subtasks. On this mission, the system immediately notices that a
step is taking far longer than usual. The system senses a problem and investigates
immediately. Sensors indicate a bolt has not been loosened (a step required for the task).
At a level of expert presentation, this minor step does not even warrant presentation (and
hasn’t been presented to this astronaut since the early days of training). The system
assumes that more information is needed and provides elementary steps and monitoring
(e.g. is tool applied, is it rotating in the proper direction). Shortly after the rudimentary
task animations, the bolt begins to loosen. Oxygen deprivation had induced erroneous
bolt-turning behavior. The error had eluded the astronaut’s detection due to her reduced
monitoring capability. Decomposition and active monitoring generated feedback to solve
the problem. The system will present more information (than it normally would), and
continue to present basic task steps until it recognizes performance has approached that
stored in her profile.
REFERENCES
Abella, B., Avarado, J., Myklebust, H. et al. (2005). Quality of cardiopulmonary resuscitation during in-hospital cardiac arrest. JAMA. 293:305-310
American Heart Association. (2005). Heart Disease and Stroke Statistics — 2005 Update. Dallas, Tex.: American Heart Association; available at http://www.americanheart.org/presenter.jhtml?identifier=1928
Anderson, J.R., (1983). The architecture of cognition. Cambridge, MA: Harvard
University Press. Anderson, J.R. (1988). The expert module. In M. Polson & J. Richardson (Eds.),
Foundations of Intelligent Tutoring Systems. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Annett, J., and Stanton, N. (Eds.) (2000). Task Analysis. New York: Taylor & Francis
Azevedo, R. & Bernard, R.(1995). A meta-analysis of the effect of feedback in computer based instruction. Journal of Educational Computing Research, 13(2), 109-125.
Balzer, W, Hammer, L., Sumner, K., Birchenough, T., Parham, S., & Raymark, P.
(1994). Effects of cognitive feedback components, display format and elaboration on performance. Organzational Behavior and Human Decision Processes, 58, 639-385. Becker, L., Eisenberg, M., Fahrenbruch, C., & Cobb, L. (1998). Public locations of cardiac arrest: Implication for public access defibrillation. Circulation, 97, 2106-2109.
Booher, H. (1975). Relative comprehensibility of pictorial information and printed words in proceduralized instructions. Human Factors, (17), 261-277.
Brown, J. S., Burton, R. R., (1978). Diagnostic models for procedural bugs in mathematical skills. Cognitive Science, 2, 155-191.
Brown, J. S., Burton, R. R., & deKleer, J. (1982). Pedagogical, natural language and knowledge engineering techniques in SOPHIE I, II and III. In D. Sleeman & J. S. Brown (Eds.), Intelligent Tutoring Systems. New York: Academic Press.
Brusilovsky, P., & Hoah-Der, S. (2002). Adaptive visualization component of a distributed web-based adaptive educational system. Intelligent Tutoring Systems, Proceedings of the 6th International Conference, Biarritz, France and San Sebastian, Spain.
Bunch, T, et. al. (2003). Long term outcomes of out-of-hospital cardia arrest after successful early defibrillation. N Engl J Med 348; 26.
Burns, H.L., & Capps, C.G. (1988). Foundations in intelligent tutoring systems: An Introduction. In M. Polson & J. Richardson (Eds.), Foundations of Intelligent Tutoring Systems. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Caffrey, S., Willoughby, P., Pepe, P., & Becker, L. (2002). Public use of automated external defibrillators. N Engl J Med, 347(16), 1242-1247.
Capucci, A., Aschieri, D., Peipoli, M., Brady, G., Iconomu, E., & Arvedi, M. (2002). Tripling survival from sudder cardiac arrest via early defibrillation without traditional education in cardio pulmonary resuscitation. Circulation, 106, 1065-1070.
Carbonell, J.R. (1970). AI in CAI: An artificial intelligence approach to computer-assisted instruction. IEEE Transactions on Man-Machine Systems, 11(4), 190-202.
ChanLin, L. (2000). Attributes of animation in learning scientific knowledge.
Journal of Instructional Psychology, 27(4), 239-243.
ChanLin, L. (2001). Formats and prior knowledge on learning in a computer-based lesson. Journal of Computer Assisted Learning, 17, 409-419.
Chi, M.T.M., Glaser, R., & Farr, M.J (Eds.). (1988). The Nature of Expertise.
Hillsdale, NJ: Erlbaum. Collins, A. (1977). Processes in acquiring knowledge. In R. Anderson, R. Spiro,
& W. Montague, (Eds.), Schooling and the acquisition of knowledge. Hillsdale, NJ: Lawrence Erlbaum Associates.
Craik, F. & Lockhart, R. (1972). Levels of processing: A framework for memory
research. Journal of Verbal Learning & Verbal Behavior, 11, 671-684.
Culley, L., Rea, T., Murray, J., Welles, B., Fahrenbruch, C., Olsufka, M., Eisenberg, M., & Copass, M. (2004). Public access defibrillator in out-of-hospital cardiac arrest. Circulation, 109, 1859-1863.
Cummins R. (1989). From concept to standard-of-care? Review of the clinical experience with automated external defibrillators. Ann Emerg Med.18: 1269–1275
Cummins, R., & Hazinski, M.F. (Eds.)(2000). Guidelines 2000 for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation. Vol. 102(8). Supplement I. August
Dix, A., Finlay, J., Abowd, G., Beale, R. (1998). Human-Computer Interaction. London: Prentice Hall Europe.
Fossel, M., Kiskaddon, R., Sternbach, G. (1983). Retention of cardiopulmonary
resuscitation skills by medical students. Journal of Medical Education, 58, 568-575.
Gleick, J. (1999). Faster: The Acceleration of Just About Everything. New York, NY: Pantheon Books.
Goodman, J. S. (1998). The interactive effects of task and external feeack on practice performance and learning. Organizational behavior and human decision processes, 76(3), December, pp.223-252.
Glaser, R. & Chi, M. T. M. (1988). Overview. In M. Chi, R. Glaser & M. Farr (Eds.), The Nature of Expertise (pp. xv-xxvii). Hillsdale, NJ: Erlbaum.
Graham, C., Scollon, D. (2002). Cardiopulmonary resuscitation training for undergraduate medical students: a five year study. Medical Education, 36, 296-298
Hinds, P.J., Patterson, M & Pfeffer, J. (2001). Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Jouran of Applied Psychology, 86(6), 1232-1243). Hallstrom, A., Ornato, J.(2004) Public-access defibrillation and survival after out-of-hospital cardiac arrest. N Engl J Med 351; 7.
Hazinski, M., Idris, A., Kerber, R., Epstein, A., Atkins, D., Tang, W., & Lurie, K. (2005). Lay rescuer automated external defibrillator (public access defibrillation) programs. Circulation, 111, 3336-3340.
Hollnagel, E. (1993). Human Reliability Analysis and Control. London: Academic Press
Huikuri, H.V., Castellanos, A., Myerburd, R. (2001). Sudden cardiac death due to cardiac arrhythmias. N Engl J Med, Vol. 345, No. 20
International Liaison Committee on Resuscitation (ILCOR) (2005). 2005 International Consensus on Cardiopulmonary Resuscitation (CPR) and Emergency Cardiovascular Care (ECC) Science With Treatment Recommendations. Circulation 112, (suppl III). American Heart Association.
Johnson, E.J. (1988). Expertise and decision under uncertainty: Performance and process. In M. Chi, R. Glaser & M. Farr (Eds.), The Nature of Expertise. (pp. 209-228). Hillsdale, NJ: Erlbaum.
Kehoe, C., Stasko, J., & Taylor, A. (2001). Rethinking the evaluation of algorithm animations as learning aids: an observational study. International Journal of Human-Computer Interaction, 54, 265-284.
Kirwan, B., and Ainsworth, L.K. (1992). A Guide To Task Analysis. London: Taylor & Francis
Klein, G. (1998). Sources of Power: How people make decisions. Cambridge,
MA: The MIT Press.
Kluger, A., & DeNisi, A. (1996). The effect of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulleting, 119(2), 254-284.
Lederer, W., Mair, D., Rabl, W., Baubin, M. (2004). Frequency of rib and sternum fractures associated with out-of-hospital cardiopulmonary resuscitation is underestimated by conventional chest X-ray. Resuscitation, Feb. 60(2), 157-62.
Lukasiewicz, J. (1994). The Ignorance Explosion: Understanding Industrial Civilization. Ottawa, Ontario: Carleton University Press.
Linn, R. L. (1990). Diagnostice testing. In N. Frederiksen, R. Glaser, A.
Lesgold, M. Shafto (Eds.), Diagnostic Monitoring of Skill and Knowledge Acquisition (pp.489-498). Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Marenco, J., Wang, P., Link, M., Homoud, M., & Estes, M. (2001). Improving survival from sudden cardiac arrest. JAMA, 285(9), 1193-1200.
Marshall, S. P. (1990) Generating good items for diagnostic tests. In N,
Frederiksen, R. Glaser, A. Lesgold, M. Shafto (Eds.), Diagnostic Monitoring of Skill and Knowledge Acquisition, pp.433-452. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc.
Miller, M., & Perlis, D. (1997). Toward automated expert reasoning and expert-
novice communication. In P. Feltovivh, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp.405-415). Menlo Park, CA: American Association for Artificial Intelligence.
Miller, R., Waitman, L., Sutin Chen, S. & Rosenbloom, S. (2005) The anatomy of decision support during inpatient care provider order entry (CPOE): Empirical observations from a decade of CPOE experience at Vanderbilt. Journal of Biomedical Informatics, 38, Issue 6 , 469-485.
Virvou, M., & Moundridou, M. (2001). Adding instructor modeling component to to the architecture of ITS authoring tools. International Journal of Artificial Intelligence in Education,12, 185-211.
Mogdaham, F. (1997). The Specialized Society: The Plight of the Indvidual in an Individualized Society. Westport, CT: Praeger Publishers.
Morgan, C., Donnelly, P., & Lester, C. (1996). Effectiveness of the BBC’s 999 training roadshows on cardiopulomonary video performance of chohort of unforewarned participants at home afterwards. BMJ, 313, 912-916.
Myerburg, R., Fenster, J., Velez, M., Rosenberg, D., Lai, S., Kurlansky, P., Newton, S., Knox, M. & Castellanos, A. (2002). Impact of community-wide police car deployment of automated external defibrillators on survival from out-of-hospital cardiac arrest. Circulation, 106, 1058-1064.
Nichol, G., Hallstrom, A., Ornato, J., Riegel, B., Stiell, I., Valenzuella, T., Wells, G., White, R., & Westfeldt, M. (1998). Potential cost-effectiveness of public access defibrillation in the United States. Circulation, 97, 1315-1320.
National Center for Health Statistics, data for 2002 (2005). National Vital
Statistics Report Vol. 53. No. 17. available at http://www.cdc.gov/nchs/data/dvs/nvsr53_17tableE2002.pdf
Naki�enovi�, N., & Grübler, A. (1991). Diffusion of Technologies and Social Behavior. Berlin, Germany: International Institute for Applied Systems Analysis (IIASA).
Palmiter, S. and Elkerton, J. and Baggett, P. (1991). Animated demonstrations vs written instructions for learning procedural tasks: A preliminary investigation. International Journal of Man-Machine Studies, 34, 687-701.
Park, O., & Hopkins, R. (1992). Instructional conditions for using dynamic visual displays: A review. Instructional Science, 21(6), 427-449.
Proctor, R., & Dutta, A. (1995). Skill Acquistion and Human Performance. Thousand Oaks, CA: Sage Publications.
Rieber, L.P. (1990). Animatin in computer-based instruction. Educational
Technoloyg, Research and Development, 38, 77-86.
Sanderson, P., Haskell, I., & Flach, J. (1992). The complex role of perceptual organization in visual display design theory. Ergonomics, 35(10), 1199-1219.
Salmoni, A., Schmidt, R., & Walter, C. (1984). Knowledge of results and motor learning: A review and critical reappraisal. Psychological Bulletin, 95(3), 355-386.
Sanders, A., & Ewy, G. (2005). Cardiopulmonary resuscitation in the real world: When will the guidelines get the message? JAMA,293, 363-365.
Sheridan, T. (2002). Humans and Automation: System Design and Research Issues. Santa Monica, CA: Wiley & Sons, Inc. in cooperation with the Human Factors & Ergonomics Society. Stanton, N., Ashleigh, M., Roberts, A., & Xu, F. (2001). Testin Hollnagel’s contextual control model: Assessing team behavior in a human supervisory control task. International Journal of Cognitive Ergonomics, 5(2), 111-123.
Starr, L. (1998). An effective CPR home learning system. AAOHN J, 46(6), 289-295.
Sternberg, R. (1997). Cognitive conceptions of expertise. In P. Feltovivh, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp.149-162). Menlo Park, CA: American Association for Artificial Intelligence. Stiell, I., Nichol, G., Wells, G., De Maio, V., Nesbitt, L., Blackburn, J., & Spaite, D. (2003). Health-related quality of life is better for cardiac arrest survivors wh received citizen cardiopulmonary resuscitation. Circulation, 108, 1939-1944.
Tversky, B. (1995). Cognitive origins of conentions. In F. Marchese (Ed.), Understanding Images: Finding Meaning in Digital Imagery, 29-53. New York: Springer-Verlag.
Teversky, B., & Morrison, J. B. (2002). Animation: Can it facilitate?
International Journal of Human-Computer Studies, 57, 247-262.
Uhr, L. (1969). Teaching machine programs that generate problems as a function
of interaction with students. Proceedings of the 24th National Conference of Education, Virginia, April 2-4, 1969. Valenzuela, T., Kern, K., Clark, L., Berg, R., Berg, M., Breg, D., Hilwig, R., Otto, C., Newburn, D., Ewy, G. (2005). Interuptions of chest compressions during emergency medical systems resuscitation. Circulation 112, 1259-1265.
Valenzuela, T., Roe, D., Nichol, G., Clark, L., Spatie, D., Hardman, R.(2000). Outcomes of rapid defibrillation by security officers after cardiac arrest in casinos. N Engl J Med, 343, 17.
Wik, L., Kramer-Johansen, J., Myklebust, H., Sorebo, H., Svensson, L., Fellows, B., & Steen, P. (2005). Quality of cardiopulmonary resuscitation during out-of-the hospital cardiac arrest. JAMA, 293, 299-304.
Willows, D. (1978). A picture is not always worth a thousand words: Pictures as distractions in reading. Journal of Educational Psychology, 70(2), 255-262.
Young, D.E. & Lee, T.D. (2001). Skill Learning: Augmented feedback. In W. Karwowski (Ed.) International Encyclopedia of Ergonomics and Human Factors. (pp. 558-562). London: Taylor & Francis.
Zipes, D., Wellens, H.(1998). Sudden cardiac death. Circulation:98:2334-2351.