Uncovering the Problem-Solving Process:
Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer
I3CLEPS Workshop/Mini-conference,
August 29, 2005
Cued Retrospective Reporting, Eye Tracking, and Expertise Differences
Overview
Experiment:
- Theory
- Design
Comparison of 3 verbal methods
The 3 methods & expertise differences
Uncovering expertise-related performance differences through eye movement data
Present limitations and future research
Discussion
Theory
Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform:
- Psychological theory
- Expert systems
- User-system interaction,
But also
- Instructional design e.g., design of process-oriented worked examples
Theory
From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000):+ of concurrent reporting (“think aloud”):
more information on actions taken+ of retrospective reporting:
more information on rationale for actions taken and strategies that control the process
Needed: A method that combines + & + :Cued retrospective reporting based on a record of eye
movements & mouse/keyboard operations?
Design
Within-subjects, 26 participants, electrical circuits troubleshooting tasks:
Seq. Condition + Tasks1 CR 1+2 CRE 3+4 RR 5+6 CRR 7+82 CRE 3+4 CRR 7+8 CR 1+2 RR 5+63 RR 5+6 CR 1+2 CRR 7+8 CRE 3+44 CRR 7+8 RR 5+6 CRE 3+4 CR 1+2
CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.
Comparison of 3 Methods: Hypotheses
1. Concurrent reporting (CR): more ‘action’ info than RR
2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR
3. Cued retrospective reporting (CRR):-> more ‘action’ than RR-> more ‘why’, ‘how’, & ‘metacognitive’ than CR
Comparison of 3 Methods: Analyses
Segmentation based on speech sentences / utterances (preceded & followed by a pause)
Coding scheme task-oriented main categories:‘action’‘why’‘how’‘metacognitive’
20% of protocols scored by 2 raters: kappa = .79 good; proceeded with 1 rater
Analyses on nr. of codes on main categories, obtained by summing codes on subcategories
Comparison of 3 Methods: Results
Friedman Tests with Conover (1999) comparisons
CR vs RR:as hypothesized: ‘action’ CR >RRhowever: ‘why’ and ‘how’ CR > RR, and‘metacognitive’ CR = RR
CRR vs RR:as hypothesized: ‘action’ CRR >RR‘why’: CRR = RR‘how’ and ‘metacognitive’: CRR > RR
Expertise Differences: Explorative
5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency:
“highest”: higher performance, lower mental effort, lower time-on-task
“lowest”: lower performance, higher mental effort, higher time-on-task
- Differences in elicited information?
- Differences in preferences/experiences?(open-ended debriefing questions)
Expertise Differences: Elicited Information
Differences in elicited information?(Mann-Whitney U Tests)
CR:
‘how’ and ‘metacognitive’ info: “lowest” > “highest”
RR:
‘why’ info: “highest”> “lowest”
‘how’ info: “lowest” > “highest”
CRR:
‘action’ and ‘metacognitive’ info: “lowest” > “highest”
Expertise Differences: Experience
Differences in preferences/experiences?“lowest”:
experience: CR (4/5)preference: CRR > CR & RR (4/5)
“highest”:no differential experiences/preferences
Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”:
- Time-on-task- Cue
Studying Expertise-Related Performance
Differences: Eye Movement Data 1
Eye fixation data provide insight in the allocation of attention, and hence differ with expertise
Research use: provide information about the problem-solving process at a finer grained level than verbal protocols?
(Ultimate) educational use: guiding novices’ attention?
1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology
Eye Movement Data: Participants & Procedure
Same 5 “lowest” and 5 “highest” expertise participants
Data collected in first 3 phases of the process:
1. Problem orientation (until pushing switch to observe circuit behavior)
2. Problem formulation and action decision
3. Action evaluation and next action decision
% time spent on phase, mean fixation duration (MFD), and in 1st phase fix. related to faults
Only 3 Volt
Short-circuit
Task
Eye Movement Data: Results
Phase 1: problem orientation(Mann-Whitney U Tests, 2-tailed, α = .10)
% of time: “highest” > “lowest”
MFD: “lowest” > “highest”
% fixations on battery: “highest” > “lowest”
Gaze switches short-circuit: “highest” > “lowest”(NB: only trend)
Eye Movement Data: Results
Phase 2: problem formulation & action decision(Mann-Whitney U Tests)
% of time: “highest” = “lowest”
MFD: “highest” = “lowest”
MFD First ½: “highest” > “lowest”
MFD Second ½: “highest” = “lowest”
Eye Movement Data: Results
Phase 3: action evaluation & next action decision(Mann-Whitney U Tests)
% of time: “highest” > “lowest”
MFD: “highest” = “lowest”
MFD First ½: “highest” = “lowest”
MFD Second ½: “highest” = “lowest”
Eye Movement Data: Results
MFD over phases (Friedman + Nemenyi post-hoc):
n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1
Limitations
- CRR and fabrication?- Cue: combination of eye movements AND
mouse/keyboard operations- Only quantitative analyses of protocols- Eye movement data: distinction of phases
- Performance efficiency measure:very relative distinction (lowest and highest within this group of participants)
- Small nr of participants in analyses related to expertise differences
Future Research
- Qualitative differences between CRR and RR?
- Cue: different effects with only eye movements OR mouse/keyboard operations?
- Cue: technical optimization?
- (RR/)CRR: effects of other prompts?
- Further study of performance efficiency measure to distinguish expertise levels
- Replications with larger N
Thank you for your attention!