The final resting place The final resting place for all this research…for all this research…

Ron Laughery, Ph.D.

University of Colorado

Items to be covered…• What is the problem this

research is trying to solve from an operational perspective?

• What is the basic human performance modeling and simulation approach that this research will feed?

• What is the specific tool and architecture that we are working to advance?

• What are the issues in moving this research into practice?

What is the problem this research is trying to solve from an operational perspective?

• The £5,000,000,000 question…– In about 1995, Robin Miller, an operational

analyst with the MoD asked us this question and made this statement at a meeting:

• “A question I get all the time can be summed up as this – should we invest £5B in new kit, or should we instead invest that £5B in training? If your models can’t help me answer that question, you’re not doing your job.”

• We are trying to ensure that we are doing our job in Mr. Miller’s eyes

What is the basic human performance modeling and simulation approach that

this research will feed?• In military and civilian systems, decisions are

increasingly being made on the basis of model based analyses– System effectiveness depends upon…

SoftwareSoftware

HumansHumansHumans

HardwareHardware

Two basic approaches to modeling human/system performance

• Reductionist– Breaking human activity and interaction

with the system into discrete activities

Advantages/disadvantages of reductionist modeling approach

• Advantages– Intuitive– Level of detail determined by need– Basic data are usually available or easily obtained– Consistent with many military systems and

operational analysis models

• Disadvantages– Often requires extensive subject-matter expert

input

Second approach to modeling human/system performance

• First principled/cognitive models– Based on theories of the underlying

mechanisms that facilitate human behavior

perception

Workingmemory

Long-termmemory

Iconicstorage

Centralprocessing

Responsemechanisms

Example: ACT-R Representation and Equations

RetrievalGoal

ManualVisual

Productions

Intentions Memory

MotorVision

World

Ai =Bi + Wj ⋅Sji +σAj∑

Bi =ln tj−d

j∑

Ui =Pi ⋅G−Ci +σU

Pi =Succi

Succi +Faili

Ti =F ⋅e−Ai

Activation

Learning

Latency

Utility

Learning

IF the goal is to categorize new stimulus and visual holds stimulus info S, F, TTHEN start retrieval of chunk S, F, T and start manual mouse movement

S 20 1Size Fuel Turb Dec

L 20 3 Y

Stimulus

ChunkBi

SSL S13

Advantages/disadvantages of the first principle approach

• Advantages– Requires less data input from either

experiments or subject matter experts– More first-principle based and, if component

models are valid, easier to defend

• Disadvantages– Model construction can be quite cumbersome

for simple tasks– We don’t have enough real first-principle

models of human performance

A strategy that has worked- a hybrid approach

• The flexibility of reductionist models combined with the power of first principles of human behavior is the formula for success

perception

Workingmemory

Long-termmemory

Iconicstorage

Centralprocessing

Responsemechanisms

Reductionist modeling with Task Network Modeling

• Largely involves the extension of a task analysis into a network defining sequencing

Going from a task network to a running computer model

• Add timing information and task/system interdependencies

Add human decision making strategies

• Any defined branch point represents a need for a decision

• Logic and rule sets, goal seeking, naturalistic

?

Then, develop a scenario, equipment model and/or links to other simulations

Run the model to collect human/system performance data

Combining First Principles of human behavior with Task Network Models

• For the past 16 years, we have been embedding and linking first principle models of human performance into our tools including– Cognitive workload and human response– Micro models of human time and accuracy– Human error and system response to error– Performance shaping factor effects– Linkage to anthropometric, biomechanical models– Goal driven task scheduling– Naturalistic Decision Making– Situation awareness modeling– Integration of cognitive engineering models such as ACT/R

• Predicting training effects is still the Predicting training effects is still the weakest link!weakest link!

Improved Performance Research Improved Performance Research Integration Tool (IMPRINT): Integration Tool (IMPRINT): Capability and ApplicationCapability and Application

What Does IMPRINT Do?What Does IMPRINT Do? It helps you... Set realistic system requirements Identify future manpower & personnel constraints Evaluate operator & crew workload Test alternate system-crew function allocations Assess required maintenance manhours Assess performance during extreme conditions Examine performance as a function of personnel

characteristics, training frequency & recency Identify areas to focus test and evaluation resources

IMPRINT Architecture - IMPRINT Architecture - Operations ModelingOperations Modeling

StochasticDiscrete Event

Simulation

StochasticDiscrete Event

Simulation

Task Time & Error Data -Estimates & Requirements

Unit &ForceUnit &Force

System MeasuresSystem

Measures

TaskLibraries

TaskLibraries

WorkloadFuture Manpower

Performance Shaping Functions- personnel

- training- stressors

WorkloadFuture Manpower

Performance Shaping Functions- personnel

- training- stressors

WorkloadFuture Manpower

Performance Shaping Functions- personnel

- training- stressors

Improved Performance Research Improved Performance Research Integration Tool (IMPRINT)Integration Tool (IMPRINT)

IMPRINT Architecture - IMPRINT Architecture - Maintenance ModelingMaintenance Modeling

Send systems on missions as defined by scenarioSimulate need for maintenance

Systems Readyfor Next Mission

Repair systems

Manpower Pool

corrective& continue

mission

combatdamage

corrective& stop

mission

preventive

Repair Parts

Who Has IMPRINT?Who Has IMPRINT?

Army Navy Air Force Other Government Contractors University

108

23

9

12

108

19

279 and growing

Mental WorkloadMental Workload

Degree of Resource Use?Which Brain Resources Involved?

Mission Tasks

1. monitoralarms

2. decideresponseaction

3. pull trigger

.

.

.n. task n

Visual

Cognitive

Auditory

Psychomotor

Cognitive

0.0 - No Cognitive Activity1.0 - Automatic (simple association)1.2 - Alternative Selection3.7 - Sign/Signal Recognition4.6 - Evaluation/Judgment (consider

single aspect)5.3 - Encoding/Decoding, Recall6.8 - Evaluation/Judgment (consider

several aspects)7.0 - Estimation, Calculation,

Conversion

Psychomotor

Auditory

Visual

Cognitive

0.0 - No Cognitive Activity1.0 - Automatic (simple association)1.2 - Alternative Selection3.7 - Sign/Signal Recognition4.6 - Evaluation/Judgment (consider

single aspect)5.3 - Encoding/Decoding, Recall6.8 - Evaluation/Judgment (consider

several aspects)7.0 - Estimation, Calculation,

Conversion

Psychomotor

Auditory

Visual

Task type (Taxon*) MOPP Heat Cold Noise Sleepless Hours

Visual T A TNumerical A TACognitive A TAFine Motor Discrete T A TFine Motor ContinuousGross Motor Light T TGross Motor HeavyCommo. (Read & Write) ACommo. (Oral) T A A

T = affects task time, A = affects task accuracy, TA= affects both

Current IMPRINT Implementation: Current IMPRINT Implementation: Stressors by Task TypeStressors by Task Type

* O’Brien, L. H., Simon R. and Swaminathan, H. (1992). Development of the Personnel-Based System Evaluation Aid (PER-SEVAL) Performance Shaping Functions. ARI Research Note 92-50

Approach to modeling human response to stressors

Performance multipliersas a function of time since sleep

6 12 18 24 30 36 42 48 54 60 66 700

0.2

0.4

0.6

0.8

1

1.2

time since sleep

performance multiplier

attention

perception

cognition

psychomotor

physical

The general effects...

Detect ring - 50% attention, 50% perception

Select menu item using a mouse - 40% attention, 60% psychomotor

Interpret customer’s request for information - 100% cognitive

On a specific task….

• attention performance multiplier = .82

• perception performance multiplier = .808

• cognition performance multiplier = .856

• psychomotor performance multiplier = .784

- physical performance multiplier = .727

under specificconditions...

leads to this specificeffect at this time…..

task time = 112.3% of normalTime to Prepare to Engage

at 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Use task network models to study aggregate effects of PSFs

Net Performance Op 6

0

2

4

6

8

10

12

14

16

18

20

100500900130017002100250029003300370041004500490053005700610065006900730077008100

Time (sec)

Wo

rklo

ad0

20

40

60

80

100

120

Uti

lizat

ion

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

Time to Prepare to Engageat 20 Hours Since Sleep

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 5 10 15 20 25

Time, sec

Relative Freq.

preptime

What view of training is in What view of training is in IMPRINT now?...IMPRINT now?...

What we really need for a reasonably What we really need for a reasonably accurate representation of training…accurate representation of training…

• We need these functional relationships…– For different task types (the taxonomy)– For different “types” of training

Performance

Amount of training received

Classroomtraining

Simulatortraining

Fieldtraining

No training Retraining

Big questions…Big questions…

• Purpose of models– Design of optimal training systems– Design of systems considering training

• Taxonomies– Training environment– Task type

• Scope/complexity of tasks studied– Do small tasks scale to large tasks?

• How do we treat Retention