CogToolA tool for interface design and ACT-R research

Bonnie E. JohnHCI Institute

Carnegie Mellon University

CogTool is an open source tool where you can describe an environment in a storyboard…

CogTool is an open source tool where you can describe an environment in a storyboard…

…and demonstrate a task

CogTool automatically creates an ACT-R model of a skilled person doing this task and produces predictions of task execution time.*

* Based on Card, Moran and Newell’s Keystroke-Level Model (1980)

And you can look under thehood to see what ACT-R is doing.

What is prediction of skilled task execution time good for? Some examples of similar analyses

• PERCs “time travel” evaluation at IBM– DARPA set requirement to show 10x productivity improvement over

2002, which we can credibly demonstrate with CogTool• Saved NYNEX from making a $160 million workstation purchase that

would have COST them $2 million/year in operating costs• IRS procurement of new IT system turned on “value” calculated, in part,

using this type of analysis (IBM lost $700 million contract to AT&T’s $1.4 billion)

• NextGen airspace will have economic consequences of time to execute cockpit tasks

• Carlsbad Police reduced injuries and loss of life in their in-vehicle information systems.

– SAE Recommended Practice J2365, Calculation of the Time to Complete In-Vehicle Navigation and Route Guidance Task.

Why should researchers at the ACT-R workshop care?

• If you are not in academia– Your organization may builds systems and this could be directly

useful for evaluating them

• If you are in academia– Consulting

• this may be a quicker way to evaluate new systems than directly coding ACT-R

– Teaching• Human Factors or User Interface Design classes and this is one

technique you could teach your students• Psychology class, part of which may be applied• Want a gentle introduction to cognitive modeling to get students

excited– Research

• Rapid environment construction• “Rapid Theory Prototyping”

• The variability between novice modelers has been reduced by 70%• This makes modeling the least variable of all usability

techniques!!!

First a word about teaching with CogToolKLMs “By-Hand” v. KLMs with CogTool (BRIMS 2010)

By-Hand KLM:Average CV=22%

CogTool KLM:Average CV=7%

Rapid Environment Construction

• Use CogTool’s storyboarding to construct an environment for your ACT-R model• Export the ACT-R code CogTool

creates• Put in your own ACT-R model

Rapid Environment Construction

• Use CogTool’s storyboarding to construct an environment for your ACT-R model• Export the ACT-R code CogTool

creates• Put in your own ACT-R model

• The interface for doing this isn’tas easy as I would like

• The environment isn’t yet a trueACT-R device model

• Anybody who would like to helpand contribute to our opensource code, please contact me

Rapid Theory Prototyping

• Or how I explored 7 theories before breakfast

Rapid UI Prototyping

UI Prototype

Prototypers = UI designers

Far easier, quicker to build than fully-functional UI

Limited to a few specific tasks

Need only be “good enough” to testuser behavior with a proposed UI

“cheating”(e.g., wizard-of-oz)

Sufficient to suggest what’s important enough to begin implementation and what should be given more thought

Tool for thought and communication

Rapid UI Prototyping Rapid Theory Prototyping

UI Prototype

Prototypers = UI designers

Far easier, quicker to build than fully-functional UI

Limited to a few specific tasks

Need only be “good enough” to testuser behavior with a proposed UI

“cheating”(e.g., wizard-of-oz)

Sufficient to suggest what’s important enough to begin implementation and what should be given more thought

Tool for thought and communication

Theory Prototype

Prototypers = Theory developers

Far easier, quicker to build than fully-functional theory

Limited to a few specific tasks

Need only be “good enough” to testtheory’s behavior against human data

“cheating”

Sufficient to suggest what’s important enough to begin implementation andwhat should be given more thought

Tool for thought and communication

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a task

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge

3. Repair the the model through rapid theory prototyping

Repeat steps 2-3 until the model is failing only by chance 4. Analyze all the failures and decide:

• Whether prototyped theory or mechanism matches human performance• If so, prioritize effort to improve the original simple theory to validate and

then move into the tool for design

Example of an aviation task

1. Start with the simplest, most generic model of a theory and ask it to do a task

Simplest, most generic model of a theory for exploring an aviation device:Information Foraging Theory (Pirolli & Card, 1999)

Augmented with the Minimal Model of Visual Search (Halverson & Hornof, 2007)

Using a general knowledge corpusEmbodied in CogTool-Explorer (Teo & John, 2008)

A task:A three step procedure to set the approach reference speed and flap angle, using the CDU in a 777.

17

Example Run of CogTool-ExplorerExample Run of CogTool-Explorer

7 theory prototypes in 4 hrsThe final one approaches human behavior (HFES09)

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

Total task

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Change knowledge (aviation vocabulary)

Elaboration + How it works knowledge

Baseline(simplest,most generic)

Different prototypes target improvement in different steps

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

Total task

1st step

2nd step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Example: Step 1

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

1st step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Step 1

Given the goal:“select landing flap and reference air speed for a approach”

Hit the INIT REFbutton

Example: Step 1

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

1st step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Change knowledge (aviation vocabulary)

Elaboration + How it works knowledge

Example: Step 1, Theory Prototype 4

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

1st step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Hierarchical visual regions & recovery

Theory Prototype 4:Hierarchical visual regions + recovery

• We “cheated” by prototyping this theory in the CogTool storyboard, not the ACT-R code

– CogTool runs ACT-R models on a UI description called a storyboard.

• States, which contain widgets (e.g., buttons), and transitions between states represent human actions on widgets (e.g., pressing a button)

• Flat representation of widgets, so visual search and information foraging considers them equally

– Cheat: Prototype visual regions = large “buttons” on the regions• When the correct region is chosen, it transitions to a state with

only the buttons in that region • When an incorrect region is chosen, it transitions to a state

with only the regions that have not been chosen– 1 hour change

Original storyboard for this task:Each state has the CDU’s 69 buttons

• How we “cheated” by prototyping this theory in the CogTool storyboard:

– Prototyping visual regions = large “buttons” on the regions

• When the correct region is chosen, it transitions to a frame with only the buttons in that region

• When an incorrect region is chosen, it transitions to a frame with only the regions that have not been chosen

– 1 hour change

Theory Prototype 4:Hierarchical visual regions

Theory Prototype 4:Prototyped as buttons in storyboard

Theory Prototype 4:Correct region buttons in region

INITREF

RTEDEPARR

ALTN VNAV

FIX LEGS HOLDFMC

COMMPROG

MENUNAVRAD

Theory Prototype 4:Incorrect region other regions

INITREF

RTEDEPARR

ALTN VNAV

FIX LEGS HOLDFMC

COMMPROG

MENUNAVRAD

No green = no cycling

Theory Prototype 4:Tremendous improvement on 1st step

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

1st step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Tremendous

improvement

Theory Prototype 4:But still insufficient to complete task

0

20

40

60

80

100

0 50 100 150 200 250

Minutes to prototype the theory

% Success

Total task

1st step

P’t

ype-1

P’t

ype-2

P’t

ype-3

P’t

ype-4

P’t

ype-5

P’t

ype-6

P’t

ype-7

Tremendous

improvement

Not sufficient to do a

multi-step task

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

1. Whether prototyped theory matches human performance2. If so, prioritize effort to improve the underlying theory

High priority: Hierarchical visual regions + recovery

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the HFES 2009 paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

High priority: Hierarchical visual regions + recovery

See Leonghwee Teo’s talk on Sunday

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

High priority: Hierarchical visual regions + recovery

See Leonghwee Teo’s talk on Sunday

General approach to rapid theory prototyping

1. Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search

2. When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery

3. Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code

Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper

4. Analyze all the failures and decide:

• Whether prototyped theory matches human performance• If so, prioritize effort to improve the underlying theory

High priority: Hierarchical visual regions + recovery

See Leonghwee Teo’s talk on Sunday

Other examples of Rapid Theory Prototyping1. My poster with Tiffany Jastrzembsk for modeling aging aduts (sort of)2. Paper at the ASSETS conference modeling blind users of screen readers