Assessment Insights from the Use of Cognitive Task Analysis to

Study Expertise Development

Richard E. Clark

Center for Cognitive Technology

Rossier School of Education

University of Southern California

The 2005 CRESST Conference

September 8, 2005

University of California, Los Angeles

Overview

• A Brief History of the Study of Expertise Development

• Cognitive task analysis, expert-novice studies, assessment and learning

• Five assessment issues derived from expert-novice studies

Brief History of Task Analysis• Social Darwinism morphed into “equal

opportunity to take test” and led to Fredric Ward’s scientific management of education

• Frank and Lillian Gilbreth (1890 to 1930’s) developed behavioral task analysis as basis for individual/team instruction and testing for best approach to tasks

• Developed QWERTY keyboard, surgical team procedures and 3x bricklaying gains

• Shiffrin and Schneider (1977), ATI research, John Anderson’s ACT* on automated knowledge led to Cognitive Task Analysis

What is Cognitive Task Analysis?

• Protocol for interviewing many experts to capture sequence, alternatives and criteria for decisions

• Retrospection by expert being interviewed while solving authentic problem

• Three to four experts interviewed and individual solution strategies are edited into one approach based on maximum efficiency and accuracy

• Range of problem examples are also collected from experts for use in instruction

• Goal is to develop a worked example of alternative or best solution strategies

What is Cognitive Task Analysis?

• Instruction and assessment largely focused on declarative knowledge – automated knowledge largely ignored in education and testing

• Decisions are largely automated, unconscious and difficult to describe during instruction

• Feldon (2004) experts 70% wrong on decisions,

• Chao (1994) w/CTA 1 expert yields 31% accuracy,3 = 75%

• Hinds (1999) experts underestimate novice difficulty

Why Task Analysis?Instruction based on CTA is significantly more effective than BTA or “conceptual analysis”

• Hoffman (1998) 50% better with CDM and textbook improvement with prenatal infections

• Velmahos et al (2002) 50% better surgical decisions, improved transfer, 25% quicker, fewer important errors

• Lee (2004) meta analysis (314 studies, 39 pre/post); Effect size of 1.79 (60% increase over alternatives)

• Kayluga & Sweller (2004) Worked examples 4x better for novices, problem solving 2x better for experts

Five Promising Suggestions

1. Focus assessment on speed, number, direction (forward or backward reasoning) and generality of hypotheses students generate

2. Early learning automates steps in solution – links between chunks of steps develop later

3. Need for secondary (timed response) measures and applications to novel versions of problems

4. Note counter-intuitive expert-novice development issues such as memory for details

5. Possible connection between misconceptions and automated routines that are difficult to change

Five Promising Suggestions1. Focus assessment on knowledge automation –

speed (secondary, timed response), number, direction (forward or backward reasoning) and generality of hypotheses students generate

2. Early learning automates decision steps – links between chunks of steps develop later

3. Need for testing of applications of solution strategies to increasingly novel versions of problems

4. Note counter-intuitive expert-novice development issues such as memory for details

5. Possible connection between misconceptions and automated routines that are difficult to change

References

Chao, Chin-Jung and Salvendy, G. (1994). Percentage of procedural knowledge acquired as a function of the number of experts from whom knowledge is acquired for diagnosis, debugging and interpretation tasks. International Journal of Human-Computer Interaction, 6(3) 221-233.

Feldon, D. (2004). Inaccuracies in expert self-report: Errors in the description of strategies for designing psychology experiments. Unpublished PhD dissertation presented to the faculty of the Rossier School of Education, University of Southern California.

Hinds, P. J. (1999). The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance. Journal of Experimental Psychology: Applied, 5(2), 205-221

Hoffman, R., Crandall, B., & Shadbolt, N. (1998). Use of the critical decision method to elicit expert knowledge: A case study in the methodology of cognitive task analysis. Human Factors, 40(2), 254-276.

Kalyuga, S and Sweller, J. (2004) Measuring knowledge to optimize load factors during instruction. Journal of Educational Psychology, 96(3), 558-568.

Lee, R. L. (2004). The impact of cognitive task analysis on performance: A meta analysis of comparative studies. Unpublished EdD dissertation presented to the faculty of the Rossier School of Education, University of Southern California.

Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E. Theodorou, D. and Maupin, F. (2004) Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. The American Journal of Surgery. 18. 114-119.