Using ACT-R for Human Error Identification

8/3/2019 Using ACT-R for Human Error Identification

1/3

Using ACT-R for Human Error Identification

Romke van der Meulen ([email protected])Department of Artificial Intelligence, Nijenborgh 9

9747 AG, Groningen, The Netherlands

Abstract

Traditional techniques for Human Error Identification mostlyconsist of hand-written, high level task models, which oftenfocus on goals and less on perception. Cognitive modellingcould avoid this problem by modelling human performance ata much lower level. This would not only rigorously identifypossible errors, but also their cause.

ACT-R is a cognitive architecture that could be used for thispurpose. It can model human behaviour and reproduce hu-man performance data. It can also reproduce human errors,by selecting incorrect production rules or retrieving incorrectchunks. It is also in a position to offer insight into the influenceof strategy on errors.

However, it would still require a human modeller to create the

ACT-R task model, before error identification can take place. Itis unclear whether automatically generated ACT-R models willbe successful in error identification. More research is needed.

Keywords: ACT-R; cognitive modelling; HEI; human erroridentification; error reproduction; strategy

Introduction

No one who cannot rejoice in the discovery of his own

mistakes deserves to be called a scholar.

Donald Foster

Errors occur every day, in any kind of situation, for any

kind of reason. To learn from our mistakes, we must

learn what went wrong. When the error occurred while

man and machine were working together, one can lookfor the cause of the error in the machine (systems ap-

proach) or in the man: this is the field of Human Error

Identification.

In the field of Human Error Identification, a great num-

ber of techniques has been developed (Kirwan, 1992).

A trait shared by many of these techniques, however, is

that they rely heavily on the insights of the modeller, and

are therefore subjective and difficult to automate. These

techniques often focus on goals, and sometimes com-

pletely disregard perceptual factors, factors often rele-

gated to the field of Human Factors research.

Some of these shortcomings could be avoided by us-ing cognitive modelling to identify human errors. These

models have the advantage of modelling human perfor-

mance very closely, and at a very low level. If such mod-

els can be used for human error identification, the results

could be quite rigorous in identifying not only possibil-

ities for human error, but also what causes such errors

and in modelling alternate approaches to the task that

might avoid these errors.

ACT-R (Anderson, 1993) is a cognitive architecture that

can be used to model human task performance. ACT-R

makes use of production rules with an IF-THEN struc-

ture. Each production rule has an associated level of ac-

tivation, which determines which rules are considered

first. Several cognitive systems, including perceptual

and motor modules, are connected to this production

system, allowing the modelling of complete tasks.

ACT-R and partial matching

At first, the implementation of the ACT-R production

system used an all-or-nothing strategy; Production rules

only fired if all their preconditions were met. This al-

lowed for errors of omission1: if the activation of a pro-

duction rule was too low it might not be retrieved (in

time). However, it did not allow for errors of commis-

sion2.

This problem was solved by allowing partial matching

within the production system (Lebiere, Anderson, &

Reder, 1994; Anderson, Reder, & Lebiere, 1996). For

each missing or differing precondition, a production rule

receives an activation penalty. Then the production rule

with the most activation is fired, even if this rule does

not fully match current conditions. Some noise is added

to the activation values of the rules, to model extraneous

influences. Under these circumstances, it can occur that

an incorrect production rule is fired, if it is close enoughto the correct rule. In this case, the model exhibits an

error of commission.

The total level of activation that can be used in ACT-

R remains constant. If, therefore, the cognitive load is

increased by, for example, a secondary task, the amount

of activation for each item drops. In this case, errors of

omission or commission are more likely, mirroring what

we find in human experiments in multitasking.

This version of ACT-R using partial matching was later

used by Lebiere, Anderson, and Bothell (2001) to model

the Air Traffic Control task. This model did well in re-

producing human performance, and even did very wellin matching the number of errors made in 8 different er-

ror categories.

1The Technique for Human Error Rate Prediction (THERP) rec-ognizes three types of error: errors of omission, where a requiredaction is not performed, errors of commission, where the wrong ac-tions is carried out or the right action is carried out incorrectly, andextraneous errors, where wrong (unrequired) actions are carried out(Kirwan, 1992)

2At least not in the production system. It was still possible for anerroneous chunk to be retrieved; more on this later.


2/3

Chung and Byrne (2008) used this modified ACT-R

model in an attempt to reproduce the post-completion er-

ror3 patterns they found in their experimental data. They

did this by introducing two rules that can fire once the

main task has been completed. One was the correct rule,

which stated that the post-completion task must first be

completed, and the other the incorrect rule, stating that areturn to the main control sequence was warranted. Us-

ing the ACT-R partial matching system, it can occur that

the incorrect rule is accidentally fired. In a second con-

dition, they added a visual cue, which is automatically

picked up by ACT-Rs visual module, leading to the ac-

tivation of additional knowledge that avoided the post-

completion error.

Using fairly standard model parameters, and without do-

ing excessive parameter-fitting, they found that the ACT-

R model produced a 14.1% PCE frequency over 200 tri-

als, where their target had been 5-15%, and that no PCE

occurred when a visual cue was presented. They con-

cluded that this was a successful application of ACT-R

as a model of their experimental data.

This shows that ACT-R with partial matching is success-

ful in reproducing not only human error rates, but even

human error types. However, in both cases the ACT-R

model was used to postdict experimental data. It remains

unclear whether ACT-R can be used to predict human er-

rors.

Errors in retrieval

ACT-R can produce errors not only through selecting an

incorrect production rule, or failing to retrieve the cor-rect one: it may also be that the production rule is cor-

rect, but the output is not. This happens when incor-

rect knowledge is present in the system in the form of

chunks, or when the wrong chunk is retrieved for a spe-

cific situation (Anderson & Lebiere, 1998).

In the first case, a chunk in an ACT-R model contains

incorrect facts. For example, a chunk may encode the

spurious fact that 1 + 2 = 4. Such a chunk can come

from several sources. It could have been explicitly cre-

ated by the modeller, in an attempt to model human error

or when trying to see how ACT-R handles the erroneous

data. It can also be the case that the chunk was createthrough the ACT-R learning process. This learning pro-

cess compiles the result of a complex production proce-

dure into a memory chunk, so that next time the result

can be retrieved without having to be reproduced. If any

of the production rules that first produced the result are

incorrect, or if noise entered during the production pro-

3A post-completion error is a special type of error of omission:it occurs when some main task has been accomplished, and the sub-

jects neglects to perform an extra required action. An often citedexample is forgetting to retrieve the original after making a photocopy.

cess, this can lead to an incorrect chunk being created.

(Anderson & Lebiere, 1998)

In the second case, it could be that the information in the

chunk is actually correct, but it is retrieved in the wrong

situation. For example, a chunk encoding 3 + 5 = 8

might be retrieved when the task at hand is to compute

3 + 4, leading to the wrong answer. This is possiblebecause the ACT-R memory retrieval system is not ex-

act: it works through association. In this example, the

chunk in question is associated to the task because both

task and chunk are in the domain of mental arithmetic,

and both relate to the concept 3. (Anderson & Lebiere,

1998)

In either case, ACT-R can learn to prevent such errors

through feedback. Whether the retrieved chunk was un-

suitable for the task or simply incorrect, through feed-

back the association between the task and the chunk can

be lowered. This will ensure that, possibly after several

trials, the association falls to the point where the chunk isno longer retrieved, and the correct chunk may be found.

Ease of use

One drawback in the use of ACT-R is that to model a

task for the purpose of error identification, the produc-

tion rules must first be found. This can be done by hand

by a human modeller, but this requires a large amount

of time and efforts, not to mention extensive domain

knowledge.

Fortunately, there are other ways to create ACT-R task

models. ACT-R can learn a task, given enough initialknowledge. However, whether such learned models will

be successful in identifying possibilities for human error

is an unanswered question.

John, Prevas, Salvucci, and Koedinger (2004) provide

another alternative. They have attempted to make the

creation of ACT-R models simple enough for usability

experts to do, by demonstrating the execution of the task

on an HTML mock-up. This is then translated into a

Keystroke-Level Model. Like the learned models, how-

ever, it remains unclear whether such generated models

will be successful in identifying possible errors.

Strategy

Another problem that all HEI techniques, including any

ACT-R approach, have to deal with, is the issue of indi-

vidual strategy. Not all users approach a task with the

same strategy. Some will make different decision from

others. This implies that users of strategy A are exposed

to different error possibilities than users of strategy B. It

is also possible for one individual to sometimes use one

strategy, and sometimes another. It is even possible for

an individual to change strategies during the task, which


3/3

often leads to a number of errors. To completely iden-

tify all possible errors, models of each strategy need to

be created, and the implications of switching strategy

understood.

Both ACT-R and traditional HEI techniques would need

to identify each strategy and model them. Parts of the

task that are the same for all strategies can be reused inthe model, but the differing parts must be individually

modelled.

It is possible for the ACT-R modelling approach to gain

an advantage in this respect. All strategies can be in-

cluded in the same ACT-R model, by collecting the pro-

duction rules of the different strategies. To model in-

dividual preferences, the activation levels of production

rules in one strategy can be heightened or lowered.

It might even be possible to model in ACT-R the effects

of an individual changing strategy at a given point in the

task. This would entail lowering the activation of the

current strategy, and heightening that of the chosen al-ternate strategy. ACT-R is in a good position to model

the kinds of errors that can occur in this situation, since

it is precisely clear what information is available at this

point, and which is lacking. The implementation of such

an ACT-R model and the possibilities it offers for er-

ror identification can make a good subject for future re-

search.

Conclusion

Human Error Identification is at present still a subjective

process. This can be improved by the use of cognitive

modelling. One possible modelling approach is the useof the cognitive architecture ACT-R. ACT-R can produce

errors of omission and errors of commission by using

partial matching in its production system. This version

of ACT-R has been used successfully to postdict experi-

mental data on different kinds of human errors. ACT-R

can also produce errors through the retrieval of incorrect

knowledge.

Unfortunately, the creation of an ACT-R task model is

an involved process, and still involves a human mod-

eller with extensive domain knowledge. There are ways

of automatically generating ACT-R models, but it is un-

clear whether such models will be successful in identi-fying (all) possibilities for human error in the task.

ACT-R is in a good position to predict errors that can oc-

cur due to a change in strategy during task execution. All

task strategies can be held in the same ACT-R model, us-

ing production rule activation levels to choose between

them. This makes a good subject for future research.

In closing, it can be said that ACT-R has a number of

advantages as a technique for human error identification.

However, practical applicability is still far off, as long as

ACT-R models must be built by hand. More research is

needed to determine whether generated ACT-R models

are successful in Human Error Identification.

It is my opinion that the use of ACT-R in Human Error

Identification is a desirable goal. It is rigorous, quantifi-

able, reproducible and produces measurable results. One

cannot model errors without modelling the task, and in

such a way that the ACT-R model can sufficiently repro-duce human performance data. This will lead to more

complete and objective models than current HEI tech-

niques can hope to attain. I therefor encourage more

research into the use of ACT-R for Human Error Identi-

fication.

References

Anderson, J. R. (1993). Rules of the mind. Hillsdale,

NJ: Erlbaum.

Anderson, J. R., & Lebiere, C. (1998). The atomic com-

ponents of thought. In (pp. 297342). Mahwah, NJ:

Psychology Press.Anderson, J. R., Reder, L. M., & Lebiere, C. (1996).

Working memory: Activation limitations on retrieval.

Cognitive Psychology, 30, 221256.

Chung, P. H., & Byrne, M. D. (2008). Cue effectiveness

in mitigating postcompletion errors in a routine proce-

dural task. International Journal of Human-Computer

Studies, 66, 217232.

John, B. E., Prevas, K., Salvucci, D. D., & Koedinger,

K. (2004). Predictive human performance modeling

made easy. In E. Dykstra-Erickson & M. Tscheligi

(Eds.), Proceedings of acm chi 2004 conference on

human factors in computing systems (pp. 455462).

Vienna, Austria: ACM Press.Kirwan, B. (1992). Human error identification in hu-

man reliability assessment. part 1: Overview of ap-

proaches. Applied Ergonomics, 23(5), 299318.

Lebiere, C., Anderson, J. R., & Bothell, D. (2001).

Multi-tasking and cognitive workload in an act-r

model of a simplified air traffic control task. In Pro-

ceedings of the tenth conference on computer gener-

ated forces and behavior representation. Norfolk, VA:

SISO.

Lebiere, C., Anderson, J. R., & Reder, L. M. (1994). Er-

ror modeling in the act-r production system. In Pro-

ceedings of the sixteenth annual meeting of the cog-

nitive science society (pp. 555559). Hillsdale, NJ:

Erlbaum.

Documents

Using ACT-R for Human Error Identification