Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Plateaus, Dips, & Leaps to Expertise
Wayne D. GrayA Presentation for the In Vivo Studies of Solo and Team PerformanceSymposium at the 2019 Conference of the Cognitive Science Society
Cognitive Science Department, Rensselaer Polytechnic Institute
1
Thanks to the Office of Naval Research!
2
CogWorks Lab – 2019: Grad Students, Post-Doc*, & Faculty
Top Row L-R: Ropa Denga, “Sasha” Lutsevich, Rousell Rahman, Chris Joanis;Mid Row L-R: Matt Sangster, Jacquelyn Berry*, Catherine Sibert, SounakBanerjee 3
Flash from the Past
What Newell Told Us
ON METHODS: First Injunction of PsychologicalExperimentation:
• Know the method your subject is using to perform theexperimental task!
• To predict a subject you must know: (1) their goals; (2) thestructure of the task environment; and (3) the invariantstructure of human processing mechanisms
4Newell, A. (1973). You can’t play 20 questions with nature and win: Projective comments on the papers of thissymposium. In W. G. Chase (Ed.), Visual information processing (pp. 283–308). New York: Academic Press
Flash from the Past – update
What We are Doing About It!
Rahman, R., & Gray, W. D. (2019). Spotlight on dynamics ofindividual learning. In Proceedings of the InternationalConference on Cognitive Modeling, ICCM
Winner of the Allen Newell Best Student-Led Paper Award for2019
5
Why & How Should We Study Human Expertise? Advice fromThorndike
“The best illustrations of mental functions at their limit ofe�ciency are to be found among those occupations of workor play [in which excellence] is sought with great zeal andintelligence” (p. 178, Edward L. Thorndike, 1913).
6Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of Learning. NYC: Teachers College, ColumbiaUniversity
Allen Newell: “Accept a single complex task and do all of it.”
60 years after Thorndike, Newell dissed, “The current experimentalstyle” of his day as trying to “design speci�c small experiments toattempt to settle speci�c small questions.”
“An alternative is to focus a series of experimental and theoreticalstudies around a single complex task, the aim being to demonstratethat one has a su�cient theory of a genuine slab of humanbehavior.”
“Unfortunately, I know of no single example which successfullyshows this scheme at work. I attribute this not to its di�culty but toits not really having been tried.” (p. 303).
Comment from Gray, Sibert, Lindstedt, and colleagues . . . We havereally tried. Newell was right, this is a very interesting andproductive approach. But Newell was also an optimist! Thisapproach is very very very di�cult with no clear ending! But it hasbeen very rewarding!!!! 7
Tetris at Home
Outline
1. Tetris at Home
1.1 Constituents of Skill in Tetris
1.2 Can Your Robot Do This?
8
Changes in Individual Performance with Expertise
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
● ●
●
●
●
●
●
−2
0
2
4
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
game difficulty level
Com
p.1
(dis
arra
y)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
−2
−1
0
1
2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
game difficulty level
Com
p.2
(4−
line
plan
ning
)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ● ●
●
●
−1
0
1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
game difficulty level
Com
p.3
(dec
ide−
mov
e−pl
aced
)
● ● ●highest (N = 27) lowest (N = 27) mid−range (N = 185)
3 components found in the Principal Component Analysis (PCA)explain just under 50% of the variance among all observedepisode-level behavioral measures of our Tetris data
9Lindstedt, J. K., & Gray, W. D. (2019). Distinguishing experts from novices by the mind’s hand and mind’s eye.Cognitive Psychology, 109, 1–25
The Disarray Component
10
Disarray is a general assessment of thedisorder in the pile, as sort of snapshotof the state of a building constructionsite.
The 4-Line Planning Component
11
4-line Planning seems less focused onimmediate behavior and more on thenature of the board structures beingbuild. This component provides anelement of higher-order cognitionwhich guides the “on board” structureas it is being built, maintained, andrepaired.
The Decide-Move-Placed Component
12
Decide-Move-Placed – our candidatemeasure of predictive processing. Thiscomponent (a) captures elements ofboth fast and accurate decision-making,(b) reveals a player’s skill across the fullspectrum of expertise represented inthis data set, and (c) across the fullbreadth of the levels of increasing timepressure.
Clark, A. (2015). Radical Predictive Processing. SouthernJournal of Philosophy, 53(1), 3–27.
Friston, K. (2018). Does predictive coding have a future?Nature Neuroscience, 21(8), 1019–1021.
The Challenge of Speed Up by Level – student players
Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7
10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 13
The Challenge of Speed Up by Level – student players
Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7
10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 14
Kill-o�at level 5
The Challenge of Speed Up by Level – student players
Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7
10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 15
Kill-o�at level 10/12
Outline
1. Tetris at Home
1.1 Constituents of Skill in Tetris
1.2 Can Your Robot Do This?
16
Can Your Robot Do Science?
• Sibert, C., Speich, J., & Gray, W. D. (2019). Less is more:Additional information leads to lower performance intetris models. In Proceedings of the InternationalConference on Cognitive Modeling, ICCM
• Sibert, C., & Gray, W. D. (2018). The Tortoise and the Hare:Understanding the in�uence of sequence length andvariability on decision making in skilled performance.Computational Brain & Behavior, 1(3-4), 215–227
• Sibert, C., Gray, W. D., & Lindstedt, J. K. (2017).Interrogating feature learning models to discover insightsinto the development of human expertise in a real-time,dynamic decision-making task. Topics in CognitiveScience, 9(2), 374–394
17
Can Your Robot Mimic Human Experts and/or Novices?
18
Can Your Robot Do Feature Learning? (CERL)
19
Tetris in the Wild
Tetris in the Wild: CTWC
20The 2018 Classic Tetris World Championships. Photography by Anthony Hornof,Professor of Computer Science and Photojournalist Extraordinaire!
Changes in Group Expertise
21
Accessing 6 years of Tetris Championship Data
Mean Qualifying Round Score by Year (95% CI)400,000
500,000
600,000
700,000
800,000
2012 2013 2014 2015 2016 2017Year
Me
an
Qu
alif
yin
g S
co
re
Changes in Individual Expertise
22
Across 6 years of Tetris Championship Data
Qualifying Round Score by Year by SID for SID >= 6 year
Chris
Matt
Vince
Trey
Harry
Alex
Robin
Jonas
Rigel
Bo
Mike
Maximum High Score -- 1,000,000
400,000
500,000
600,000
700,000
800,000
900,000
1,000,000
2012 2013 2014 2015 2016 2017Year
Me
an
Qu
alif
yin
g S
co
re
If Only We Had a Time Machine!!!
Ah, but we do!!
23
If Only We Had a Time Machine!!!
Ah, but we do!!
23
Post-hoc Longitudinal Studies
24
Data out of 6 years of Tetris Championship Videos
Natural Experiments: What we are learning from the Game In-ventions of the Tetris Masters – NO NEXT BOX TETRIS!!
25
now is the time
Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7
10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100
26
Our championsbeginat level 18!
Was Newell Right?
Is the right way to advance cognitive science to take acomplex task and do all of it??
Was Newell naive!! Maybe.
We thought we had learned a lot about Tetris expertise fromour machine modeling and from the 240 undergraduates whoplayed Tetris in our lab. We had but . . . the Tetris masters hada lot more we could learn. Our champions begin at level 18 –far beyond our best ugrad.
27
28
Thank You!!Two excellent students on the job market! One istied to the New York City/Rutgers/Princeton area.The other is not. Please send me [email protected] for more info.