Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

1/42

Multitasking & Workload

Felix Putze 14.6.2012

Lecture „Cognitive Modeling“ SS 2012

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

2/42

Multitasking • In many real-world applications, people deal with multiple tasks

in parallel (e.g. driving and operating cell phone) • Multitasking is both a chance for increased efficiency but also a

thread of degrading performance • Important questions from a HCI perspective:

• How are multiple tasks handled by the human (e.g. real parallelism vs. sequential processing)

• Which tasks can be executed in parallel? • How does multitasking

influence performance? • How can multitasking be modeled?

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

3/42

Terminology • Task: Combination of an objective, actions and external stimuli • Dual Task: Two different tasks are executed in parallel

• Multitasking: Two or more tasks

• For complex scenarios, identification of tasks may be not clear • Car driving can be regarded as one compound task or as a combination of

several simpler tasks (e.g. keep lane, manage distance to other cars, …)

• Often, two dual tasks are not equally important • Primary Task: Main task a person is concentrating on • Secondary Task: Distraction task which has to be executed in parallel • Example: Driving a car (primary) and using a mobile phone (secondary)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

4/42

Example: Schumacher Task • Dual task with equal priorities (Schumacher, 2001) • First (visual-manual) task:

• Circle appears on screen in one of three different positions • Press button according to the position

• Second (aural-vocal) task: • One of three different tones (different in pitch) is played • Say “one”, “two” or “three” according to the tone

• For the dual task condition, visual and aural stimuli are presented in synchronized fashion

• Task describes a sequence of execution conditions: • Day 1: Training of single tasks • Day 2-5: Execution of single tasks and dual task condition

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

5/42

Observations for Schumacher Task • After a training period, people achieve perfect time sharing

• No overhead in processing time of parallel stimuli

• Variations of the task: • Give priority to the aural-vocal task and present visual stimulus with a

temporal offset no perfect time-sharing observed anymore − Explanation: Perceptual Refractory Period, i.e. processing of first

stimulus (priorized) not finished when second stimulus arrives • Make visual task harder by a less intuitive stimulus-key mapping no

perfect time-sharing observed anymore − Explanation: In original task, visual task was processed faster than

aural task, i.e. no overlap in response generation

• Perfect execution of two parallel tasks is possible but requires learning and depends on several context factors

• What factors influence the difficulty of executing two tasks in parallel?

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

6/42

Influencing Factors: Practice • Frequent repetition makes parallel execution of tasks simpler

• Example: Compare expert and novice car drivers holding a conversation • See also the Schumacher task

• Automatic vs. controlled processes • Controlled: requires attention, of limited capacity, can be used flexibly,

scheduled by flexible high-level attentional system • Automatic: no attention, no capacity limit, hard to modify once learned,

scheduled by simple contention scheduler based

• Practice transforms controlled processes to automatic

processes and supports parallelization • Evidence from fMRI data

• During training, activation in central executive of working memory is reduced over time

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

7/42

Automating theories • Possible explanations of automation:

• Gradual change: Optimization of existing solutions • Restructuring: Finding shortcuts or new ways of solving the problem • Example: Calculating 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3

− Gradual: Optimization of fast counting/addition − Restructuring: Calculating 10 * 3

• Instance Theory (Logan, 1988): Automaticity = Memory • Memory traces are stored when a stimulus is perceived and processed • Training reinforces those memory traces increasing knowledge base • “Performance is automatic when based on single-step direct access

retrieval of past solutions” • When no practiced solution is available, thought and attention are

required

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

8/42

Influencing Factors: Difficulty • Harder tasks are harder to perform in parallel

• Harder task require more attention/cognitive resources • Easier tasks can be automated more easily

• Again, it is difficult to precisely measure difficulty in general • Often, difficulty(dual task) > difficulty(task1) + difficulty(task2)

• Additional coordination overhead • Example: rotating right hand clockwise, rotating right foot counter-

clockwise Easy to do sequentially, hard to perform in parallel • Evidence from fMRI data (D’Esposito, 1995): Additional activity in

prefrontal cortex and anterior cingulate only in dual-task condition responsible for conflict resolution and decision making

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

9/42

Influencing Factors: Similarity • Tasks which are similar to each other are harder to execute in

parallel than more distinct ones • Similarity can be measured using several simple criteria:

• Input modality: aural, visual, haptic, … • Response type: vocal, manual, …

• In general, similarity of two complex tasks is hard to measure • How similar are piano playing and poetry writing compared to playing

chess and boxing? • We may require a detailed model of the involved cognitive processes

to estimate similarity

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

10/42

Central Capacity Theories • Kahneman (1973): Theory of graduated effort

• Multiple tasks compete for abstract cognitive resources • A fixed pool of such resources is available • Conflicts between tasks occur when more resources are

concurrently requested than available

• Kahneman’s model explains… • …why easier tasks can be parallelized more easily • …why learning helps (as it makes execution of tasks easier)

• However, it does not model why similar tasks are hard to execute in parallel

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

11/42

Multi-Resource Theories • Wickens (1984): Multi-resource model

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

12/42

Dimensions of Wickens’ Model (1) • Wickens’ multi-resource model defines four dimensions which

categorize a cognitive task • Stages: General steps of information processing

• The perceptual and cognitive stage share the same resources (e.g. working memory)

• The responding stage does not interfere with those

• Perceptual modalities: On which channel(s) do stimuli get from the external world to the cognitive system? • Visual: Colors, Shapes, Position, Symbols, … • Auditory: Sounds, Speech, …

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

13/42

Dimensions of Wickens’ Model (2) • Visual processing: Finer differentiation of visual perception

• Focal: reading text, pattern recognition, … • Ambient: sensing orientation and ego motion, uses peripheral vision • Distinction explains ability to walk while reading a book

• Processing Codes: How is the processing information represented mentally? • Symbolic: Linguistic, mathematical, … • Spatial: Location, mental rotation, … • Also deals with response generation

− Manual: Button pressing, writing, … − Vocal: Saying answer

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

14/42

Multi-Resource Theories • Can be used to derive a simple task demand model

• Classify each task as automated (D = 0), easy (D = 1), or difficult (D = 2) • Count the number C of dimensions for which two tasks overlap • Task demand = D + C

• Rationale behind the selected dimensions: • Neurophysiologic Plausibility:

− auditory and visual cortex − left and right hemisphere (spatial vs. verbal) − motor cortex − …

• Helpfulness for human factor analysis: Direct translation from dimensions to aspects of a system: − keyboard vs. voice control? − Output graphs or digits? − In synthesized voice or on screen? − …

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

15/42

Serial Bottleneck • ACT-R: Production system imposes a cognitive bottleneck

• Only one rule can be executed at a time (i.e. every 50ms) • Production rules modeling different tasks are executed in sequence • Called the “serial bottleneck” • No perfect time sharing between different tasks possible

• Other architecture (e.g. EPIC) do not have this bottleneck • EPIC can fire any number of production rules in parallel • Natural model of multitasking

• ACT-R is the more constrained architecture • Requires more attention to model multitasking with perfect time-sharing • Able to model multitasking situations with degrading performance • fMRI data (Graybiel & Kimura, 1995) indicates suppression of other

potential rules by the best-matching one

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

16/42

Threaded Cognition in ACT-R • How to model multitasking considering the serial bottleneck? • Threaded Cognition started a custom expansion to ACT-R

modeling multitasking by Salvucci and Taatgen (2008) • Now included in the standard ACT-R distribution

• Transparent extension of ACT-R mechanisms to support

multitasking scenarios • Each task is considered to be a thread in ACT-R • A thread consists of a sequence of production rules related to one goal • Threads compete for cognitive resources

• Evaluated against a number of examples

• Multiple cognitive tests with different characteristics • Real-world scenarios like driving with a secondary task

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

17/42

Multiple Goals • Vanilla ACT-R maintains exactly one active goal at a time • For multitasking, we have several goals which correspond to

different tasks/threads • Extend goal buffer to contain a set of active goals

• One production rule (e.g. one which triggers a dual task) can put multiple goals into the buffer

• For cognitive resources (central executive, declarative module, perceptive and motor components), we keep the assumption of serial access • Note that this is stricter than Wickens’ model which allows parallel

access as long as enough resources are available

• Vanilla production rule selection still elects and executes one rule at a time

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

18/42

Resource Allocation • Threads allocate resources in a greedy and polite manner

• Greedy: Request as early as needed (no earlier, but also no later) • Polite: Release resources as soon as result is retrieved

• When multiple threads contend for the procedural resource, the least recently processed thread is allowed to proceed • Simple load balancing mechanism • Allocation of other resources can only happen if a production rule is

fired procedural module is single contended resource

• If a required resource is busy, the thread implicitly waits (i.e. cannot execute production rules)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

19/42

Schumacher Task with Threaded Cognition • Start with two vanilla models of each task

• No explicit rules for multitasking necessary • Some common rules can be shared between both tasks

• Insert initial goals of both tasks into the goal buffer • Reminder: ACT-R learns new production rules by

concatenating existing ones • Models transition from controlled to automatic execution

• Untrained model must retrieve production rules for every task step • Untrained model must retrieve stimulus-response mappings from

declarative memory • Trained model solves a task with just one production rule retrieval and

execution

• Model reflects training process of time sharing

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

20/42

Novice Behavior • Many idle blocks

• Caused by frequent access to declarative memory

• mappings from stimuli to responses have to be recalled

• Low degree of parallelism • overhead increases

processing time

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

21/42

Expert Behavior • Makes use of specialized

rules learning during training

• No explicit retrieval of mappings from memory possible anymore

• Model exposes nearly perfect time-sharing

• Models can also be modified to reflect the impact of the experiment variants

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

22/42

Implications for HCI • Users are able to perform multitasking

• Artificially restricting them to one task at a time reduces efficiency

• Multitasking performance can degrade • Forcing multiple tasks to be executed in parallel can impact performance

• Subtle changes in the task formulation can lead to very different outcomes

• A model of multitasking allows to determine… • … whether multitasking leads to performance degradation • … which resources are the bottlenecks of the dual task • Wickens’ multi-resource model helps, but only allow static analysis (does

not take time course into account)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

23/42

Example: Distract-R • Rapid bootstrapping tool for in-car

interfaces 1. Design GUI using a graphical editor 2. Select parameters of driver model

and driving scenario 3. Run simulation (using Threaded Cognition

in ACT-R) 4. Inspect Results

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

24/42

Workload • Most dual tasks require enormous “mental effort” • The “mental effort” required for a task determines how well it

can be executed and whether parallel tasks are possible

• Cognitive workload influences human cognition and behavior • Memory span and capacity may be reduced • Attention may be limited • Actions are may be executed less precise • General task performance may degrade • …

• Need models and adaptive systems to represent the concept of workload for realistic models in complex scenarios

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

25/42

Terminology • Task Complexity: Amount of mental resources required by a

certain task • Workload: “The perceived relationship between the amount of

mental processing capability or resources and the amount required by the task“ (Hart & Staveland, 1988) • Contains a subjective assessment • May depend on individual parameters like general intelligence, skill or

current vigilance

• In line with Wickens’ multi-resource model, we sometimes define different types of workload: • Cognitive workload • Visual workload • Physical workload • …

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

26/42

Measuring Cognitive Workload: Behavior • As the task demand increases, people will make more errors

• Validated hypothesis: high workload many errors • Approach for WL estimation: many errors high workload • Number and severity of errors are individually different

− More flexible WL measure than a-priori task demand • However, errors can be caused by other factors, for example:

− Inattention − Inexperience − Misunderstanding of task

• Need normalization for individual baseline performance • We measure the actually observable and measurable impact

of workload ( objective criterion) • If a person is overloaded, the number of errors might be high

although the actual workload is low (person has given up)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

27/42

Measuring Cognitive Workload: Physiology • Many physiological parameters react to increasing workload • Cardiovascular System

• Heart rate increases • Heart rate variability increases

• Respiration • Respiration rate increases

• Perspiratory glands • Skin conductivity increases

• Brain activity • Relaxation rhythms disappear:

− α-rhythm in frontal cortex − μ-rhythm in motor cortex

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

28/42

Workload Recognition • Use physiological measurements to generate an empirical

model of workload • Learn mapping between features extracted from biosignals to a

number of workload states

• Potential applications of model: • Integrate online model into an interaction system for adaptation • Provide information on workload to other cognitive models to change

their behavior (e.g. decreasing memory capacity if workload increases) • Use workload measure for usability evaluation of systems (e.g. identify

notoriously difficult parts of the interface)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

29/42

Workload Recognition: Example Application • Empirical workload model can be

employed by technical systems to adapt their interaction strategies

• Example: Robot with information- presentation task

• Changes presentation style based on the detected workload (low or high):

• Evaluation shows that adaptive strategy outperforms static behavior in efficiency, effectiveness and user satisfaction

LOW WORKLOAD HIGH WORKLOAD The name of the next person is Heidi Kundel. Her telephone number is 52-11-66-3

Heidi Kundel Telephone: 5-2-1-1-6-6-3

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

30/42

Measuring Cognitive Workload: Subjective Criteria • Subjective experience of workload can differ from objective

measurement • Empirical evidence indicates that relation between objective and

subjective workload is governed by a power law (like loudness, brightness, …) no linear relationship

• Experienced workload depends on many individual factors

• Important criterion for user satisfaction • Measured usually with post-hoc questionnaires

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

31/42

NASA TLX • Multidimensional workload questionnaire • Design goal: Reduce inter-personal variation due to individual

definitions of which factors contribute to subjective workload • Identify six contributing dimensions from experiments:

• Mental demand • Physical demand • Temporal demand • Performance • Effort • Frustration level

• Subjects rate each subscale from 0 to 100 • Have subjects create pair-wise importance ratings of factors

• Results in weight for each subscale

• Subjective Workload = Weighted sum of subscales

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

32/42

Workload Profile • Multidimensional workload questionnaire • Closely related to the multi-resource model • Useful for evaluation purposes in HCI

• Identify cognitive resources which are most busy • Improve components which load on those resources

• High sensitivity for different task difficulties and task types • High diagnosticity (i.e. yields explanations for observed scores) • Drawback: Requires extensive explanations for the users to

understand the very abstract dimensions of the questionnaire

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

33/42

Modeling Effects of Workload • Ideal: Describe whole activity of a human in one joint model Architecture itself implicitly models effects of workload, e.g. • Limited resources, e.g. memory capacity • Bottlenecks in multitasking situations, e.g. exclusive resources • No need for an explicit representation of workload

• This approach is problematic in realistic scenarios: • The employed architecture might not be designed to implicitly model all

effects of increasing workload • We might not have modeled all tasks which cause workload

(Perhaps we are only able to use empirical models to estimate workload)

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

34/42

Overlays and Modulators • Overlays are components which modify behavior of the core

architecture based on certain input variables • Eyeglass overlay: Increase number of visually processable items • Caffeine overlay: Improve processing speed by 5%

• Example: Overlays to model stress • Reduce range of visual perception ( cognitive tunneling) • Increase retrieval threshold of declarative module • Increase speed of decay in declarative module • Create a secondary “worry” task

• Architecture Co-Jack: Designed to contain a layer of modulators (≈ overlays) as main design principle

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

35/42

Model Human Processor • Model Human Processor (MHP)

• Cognitive Modeling method to predict duration of task processing

• Defines several cognitive processors − Perceptual − Cognitive − Motor

• Each processor has individual processing time to perform one operation

• For HCI, describe a task in pseudo code in terms of those cognitive processors calculate task processing time

• Queuing Network Model Human Processor (QN-MHP) • Transfers the MHP model to a queuing network architecture • Can be simulated using standard queuing network modeling toolkits

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

36/42

QN-MHP: Architecture

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

37/42

QN-MHP: Servers • QN-MHP consists of servers in a queuing network

• Servers provide functionality to customers • Server has a delay and a waiting queue for customers • Output of one server is input of its successors • Established modeling tool for transportation or communication

• The three main components of MHP are now sub-networks • Server definitions are inspired by…

• Neuro-scientific findings (e.g. visual subnetwork, nodes 1-4)

• Cognitive models (e.g. Baddeley’s memory model, nodes A and B)

• A server has a probabilistic processing time • Exponentially distributed • Parameters chosen to reflect the behavior of the original MHP

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

38/42

QN-MHP: Workload Estimation • Goal: Predict NASA TLX scores from network simulation • Idea: Average utilization of a sub-network = index of

subjective mental workload • Motivation: Increasing utilization of certain brain regions

results in increased neurotransmitter consumption perceived as mental fatigue

• We have • ρi(t) is the average utilization of sub-network i at time t

• We then define for example mental demand (MD) as depending on perceptual and cognitive sub-networks:

• Analog definitions for other dimensions

TdtT

ii

= ∫0 ρρ

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

39/42

QN-MHP: Example of Multitasking Behavior • Model performs dual in-car task:

• Lane-keeping on a curved road • Map reading on a display

• For evaluation, the QN-MHP model was connected to a driving simulator and controlled its steering

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

40/42

QN-MHP: Evaluation of Multitasking Behavior • Model was evaluated by comparing its performance to the

performance of humans • Single-task performance:

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

41/42

QN-MHP: Evaluation of Multitasking Behavior • Dual-task performance:

• Evaluation indicates that the model is able to predict the average human behavior in both conditions

Cogn

itive

Mod

elin

g: M

ultit

aski

ng &

Wor

kloa

d

42/42

QN-MHP: Evaluation of Workload Prediction • Model can predict workload over time based on utilization of

subnetworks:

• Predicted NASA TLX scores where compared to those from human subjects: