Windows Into The Mind
Richard J. Shavelson & Maria Araceli Ruiz-PrimoStanford University
Invited TalkUniversity of Ancona
Ancona, ItalyJune 26, 2000
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
2
DENNIS THE MENACE
“GEE! I’M GLAD WE DID CONCEPT MAPS - NOW WE BOTH KNOW WHAT I LEARNED TODAY!!! ”
Assessing Knowledge Structure
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
3
Overview
• Cognitive theory underlying assessments of knowledge structure
• Direct and indirect methods of assessing “cognitive structure”
• Direct method: Concept maps• Indirect method: Cognitive maps• Link between concept and cognitive maps• Concluding comments
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
4
Toward An Achievement Framework:Knowledge Types and Dimensions
Extent(How much?)
Structure(How is it organized?)
Others(Precision? Efficiency? Automaticity?)
Declarative Procedural StrategicKnowledge Knowledge Knowledge(Knowing the “that”) (Knowing the “how”) (Knowing the “which,” “when,” and “why”)
CognitiveCognitiveTools:Tools:PlanningPlanning
MonitoringMonitoring
Domain-specific content: • facts• concepts• principles
Domain-specificproduction
systems
Problem schemata/strategies/
operation systems
ProficiencyLow High
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
5
is an example of
Associative and Semantic Models of Memory
• Nodes represent concepts
• Lines represent relations among concepts
• Probe memory by asking about concepts (or lines) Associative Semantic
is a
contains
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
6
Concepts Maps
• Methods (Mapping Techniques)– Constructing A Map– Fill-In-The-Node Map– Fill-In-The-Line Map– Many Others
• Data structure and scoring methods depend on the technique
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
7
What Are Concept Maps?A Concept Map of a Concept Map!
graph
ConceptConcept MapMap
consist of consist of
is a
nodes labeledlines
concepts relation
denote arepresent
linked by
between a pair of
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
8
Concept Map Assessment Components
• TaskTask that invites a student to provide evidence bearing on his or her declarative knowledge structure in a domain
• FormatFormat for the student’s responseresponse
• Scoring systemScoring system by which the student’s concept map can be evaluated accurately and consistently
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
9
Are There Different Types of Concept Maps?
Yes, more than we would like!Yes, more than we would like!
Variations in the task, the response format, and the scoring system produce different concept
mapping techniques.
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
10
Concept Map Assessment: An Example
The following map shows the relationship among concepts related to genetic continuity. The concepts are arranged hierarchically and linked each other. Please examine the map and supply, in the space provided, a word or two for labeling each link such that association between concepts is made clear.
TaskTask
Response FormatResponse FormatGenetic
continuity
Asexualreproduction
Sexualreproduction
Meiosis
Malegamete
Femalegamete
1. through2. producer3. requires4. called5. fuse into
Scoring SystemScoring System
• One point is given for each correct relationship
• One point is given for each level up until two levels beyond the last branching
• One point is given for the first branching where two or more concepts are connected to the concept above. Three points are given for any subsequent branching
• One point is given for each cross link
Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements,learning processes, and prior knowledge. Boston: Kluwer Academic Publishers.12
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
11
Another ExampleTaskTask
Response FormatResponse Format Scoring SystemScoring System
Construct a concept map that reflects what you knowabout Ions, Molecules, and Compounds. The map shouldhave 10 concepts in it. Organize the terms in relation to one another in any way you want.
Focuses on the accuracy of the propositions:• Four points if the proposition is
outstanding and complete• Three points if the proposition is
correct and complete• Two points if the proposition is
incomplete but correct• One point if the proposition is correct
but does not show understanding between the two concepts.
• Zero if the proposition
Blank page for constructing the
map
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
12
On The Reliability & Validity of Concept Maps
• ReliabilityReliability: Can a student’s concept maps be consistently scored by different persons?
• Equivalence of Mapping TechniquesEquivalence of Mapping Techniques: Do all mapping techniques provide the same picture of a student’s declarative knowledge structure?
• Task DemandsTask Demands: Do different mapping techniques impose different cognitive demands on a student?
• Comparing Concept Maps With Traditional AssessmentsComparing Concept Maps With Traditional Assessments: Do concept maps measure something different than multiple-choice tests?
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
13
Study 1:Tasks
No-Concepts Provided Concepts Provided
• Construct a concept map about Ions, Molecules, and Compounds.
• Select another 7 concepts that you think are the most important in explaining the topic.
• Organize the concepts
• Construct a concept map about Ions, Molecules, and Compounds.
• Examine the 10 concepts and construct the map.
• Organize the concepts
• Redraw the map so someone else can read it.
Mapping Technique 1 Mapping Technique 2
LIST OF CONCEPTSacidsanions ...
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
14
Response Format
Blank page to construct the map
Construct-a-Map: No Concepts
Construct-a-Map: Concepts Provided
Blank page to construct the map
Concepts: Atoms, Compounds, Ions...
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
15
Scoring System
• Proposition Accuracy:Proposition Accuracy: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent
• ConvergenceConvergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map
• Salience:Salience: Proportion of valid propositions to the total number of propositions in student’s map
Based on a criterion map, scoring focuses on:
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
16
Proposition “Quality”Quality ofProposition Definition
Outstanding proposition. Complete and correct. It shows a deepunderstanding of the relation between the two concepts.acids-compounds: < that gives off H+ when dissolved in water are
Excellent 4 points
GoodComplete and correct proposition. It shows a good understandingof the relation between the two concepts.acids-compounds: > are examples of
3 points
Poor Incomplete but correct proposition. It shows partial understandingof the relation between the two concepts.acids-compounds: < form
2 points
Don’t CareAlthough valid, the proposition does not show understandingbetween the two concepts.acids-compounds: > is a different concept
1 points
Invalid Incorrect proposition.acids-compounds: > made of
0 points
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
17
Generalizability(Reliability) of Scores
Percent of Total Variability and G Coefficients for Proposition Validity Total Scores
No-Concepts,Sample A, and B
Sample A & B
Person (p)Rater (r)Sample (s)p x rp x sr x cprc,e
2
71.64 78.67 .15 0 0 0 0 .7922.81 17.64 .01 .18 5.37 2.69
.89 .88 .89 .88
.78 .78
2 (nr=2,ns=3)
(nr=1,ns=1)
(nr=2,ns=2)
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
18
Discriminant Validity
Type ofScore No-Concepts Sample A Sample B
Proposition Validity
Congruence
Salience
Correlation Between Multiple-Choice Test and Concept Map Scores
.58
a
.45
.64
.66
.61
.63
.55
.50
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
19
Study 2:Tasks
Construct-a-Map Fill-in-the-Nodes Fill-in-the-Linking Lines
• Construct a map.
• Use the 20 concepts provided to construct the map.
• Organize the concepts any way you want.
• Check the map.
• Redraw the map so someone else can read it.
• Examine the skeleton map, the blank nodes and the terms provided.
• Select the term that corresponds to each blank node.
• Write the term inside the circle.
• Check that all blank nodes are filled-in.
• Examine the skeleton map, the blank linking lines and the linking words provided.
• Select the linking words that correspond to each blank linking line.
• Write the linking words on the blank linking line.
• Check that all blank lines are filled-in.
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
20
Response Format
Construct-a-Map Fill-in-the-Nodes
atoms
compounds
N2O4
Elements, Atoms, and Compounds
Fill-in-the-Linking Lines
Elements, Atoms, and Compounds
formed with cations
and anions areformed frommolecules sharing
example of
made upof opposite
composed of
that begins with H are
lose or gainelectrons to
form
with 3 or moreelements are
atoms
compounds
N2O4
made upof opposite
composed of
with 3 or moreelements are
ions
ioniccompounds
molecularcompounds
ternaryionic
acids
formed frommolecules sharing
Blank page to construct the map
Concepts: Elements, Atoms, Compounds...
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
21
Construct-A-MapConstruct a concept map that reflects what you know about what a concept map is. Organize the terms in relation to one another in any way you want.
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
22
Fill-In-The-NodeExamine the map, the blank nodes and the terms provided in the list below. Select the term that corresponds to each node and write it down inside the circle. Use each term only once.
Compounds
ELEMENTS, ATOMS, AND COMPOUNDS
Atoms
are composed of the same type of
Periodic Table
are organized on a
with equal number of electrons and protons have a neutral
have a negative
are negatively charged
particles of
have a neutralhave either a
positive or a negative are formed when
atoms lose or gain
are combined chemically
to form
made up of opposite charged
lose or gain electrons to
form
are composed of more than one type of
is a ternary ionic compound that contains sulfite
which is a
are located on the left side of the
with 3 or more elements are
are elements located in the center of the
containing 2 or more elements are called
is an example of a
contain one or more
with two elements are
are composed of cations and
have to have hydrogen
that begin with H are
are composed of H+ and
are composed of metals and
tend to form
tend to form
are located on the upper right side
of the
have a positive
have a negative
form ions that can
have different
will form
is an example of a
are atoms with more protons than electrons and are
called
are atoms with more electrons than protons
and are called
containing molecules
and sharing electrons are called
that are formed from the attraction of cations and
anions are called
A
are composed of two
Transition Metals
CationsPolyatomic Ions
N2O4
Electrons
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
23
Fill-In-The-Linking-LinesExamine the map, the blank lines and the linking words provided on the list below. Select the word(s) that corresponds to each line and write the number down in the line. Use each number only once.
Compounds
Molecular Compounds
Anions
Ternary Ionic Compounds
ELEMENTS, ATOMS, AND COMPOUNDS
Atoms
Elements
are composed of the same type of
Periodic Table
are organized on a
are negatively charged
particles of
have either a positive or a
negative are formed when atoms lose or gain
are combined chemically
to form
made up of opposite charged
are composed of more than one type of
is a ternary ionic compound that contains sulfite
which is a
are located on the left side of the
are elements located in the center of the
contain one or more
with two elements are
have to have hydrogenare composed of
H+ and
are composed of metals and
tend to form
are located on the upper right side
of the
have a positive
have a negative
will form
Metals
containing molecules
and sharing electrons are called
C
are composed of two
Transition Metals
Non-Metals
Charge
Ions
Polyatomic Ions
Cations
Electrons
Ionic Compounds
N2O4
Sodium Sulfite
Binary Ionic Compounds
Acids
is an example of a
with equal number of electrons and protons have a neutral
have a negative
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
24
Construct-a-Map Fill-in-the-Map
Scoring System
• Proposition Accuracy:: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent
• Convergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map
• Salience:: Proportion of valid propositions to the total number of propositions in student’s map
• Correct/Incorrect: Student’s responses on the skeleton map were scored as correct or incorrect.
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
25
Are Fill-In-The-Map Scores Sensitive To The Nodes/Linking Lines Selected To Be Filled-In?
Type of Skeleton Map Mean S.D.
Means and Standard Deviations by Type of Map and Sample
Fill-in-the-nodes
Sample 1 80 11.21 1.42
Sample 2 72 10.80 1.74
Fill-in-the-linking lines
Sample 1 78 9.77 2.74
Sample 2 73 8.99 3.09
(Max = 12)n
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
26
Are The Two Forms Of Fill-In-The-Map Equivalent?
SequenceFill-in-the-nodes Fill-in-the-linking lines
Mean S.D. Mean S.D.
Means and Standard Deviations by Type of Map and Sequence
1 Nodes 1-Lines 1 43 11.09 1.52 9.72 2.84
2 Nodes 1-Lines 2 36 11.03 1.33 9.31 3.06
3 Nodes 2-Lines 1 35 10.63 1.81 9.83 2.65
4 Nodes 2-Lines 2 37 10.97 1.67 8.68 3.13
Total 152 11.02 1.59 9.36 2.93
n
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
27
Can The Two Mapping Techniques Be Considered Equivalent?
Type of Assessment n Max Mean S.D.
Means and Standard Deviations across the Three Types of Assessments
Construct-A-MapProposition Accuracy 152 135 53.91 22.17Convergence 152 1 .50 .19Salience 152 1 .73 .17
Fill-InFill-in-the-nodes 152 12 11.02 1.59Fill-in-the-lines 151 12 9.39 2.93
Multiple-Choice Test 150 30 24.05 3.74
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
28
Do The Mapping Techniques Provide Information Similar To That Provided By A
Multiple-Choice Test?
Type of Assessment
Construct-A-Map (C-M) (.99)Fill-in-the-nodes-NOD (FI-N) .47 (.71)Fill-in-the-lines-LIN (FI-L) .44 .40 (.85)
Multiple-Choice Test (MC) .44 .37 .53 (.74)
C-M FI-N FI-L MCStructure Extent
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
29
Other Empirical Evidence
• Correlations tell you about rank ordering on different measures--that’s only part of the story
• Evidence is needed for the cognitive claims that different methods measure somewhat different aspects of achievement:– Talk aloud– Focus group– Group work– Other?
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
30
Talk Aloud with Concept Maps: Method Variation
0
0.1
0.2
0.3
0.4
0.5
0.6
Explanation Monitoring ConceptualErrors
No-Code
Contruct-A-MapFill-In-NodesFill-In-Lines
Verbal Codes
Prop
ortio
n Sc
ores
Correspondence Between Directedness and Inferred Activity
Low Directed
High Directed
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
31
Tentative Concept Maps Conclusions • Students’ maps can be consistently scored by different persons even when
complex judgments, such as proposition quality, are required
• Different mapping methods provide different pictures of students’ declarative knowledge structure. The construct-a-map method provides opportunities to reveal students’ conceptual understanding
• Different mapping techniques impose different cognitive demands on students. Highly structured mapping techniques allow students to respond by elimination or guessing, whereas constructed response do not.
• Correlations between concept-map and multiple-choice scores are positive and moderate suggesting that these two types of assessment measure overlapping but somewhat different aspects of declarative knowledge
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
32
Cognitive Maps
• Methods– Word Association– Similarity Judgment– Card Sorting– Tree Building
• Data structure: Proximities or distances• Scaling methods: Networks or hierarchical
clusters
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
33
Example: Similarity Judgments
Rate the similarity of each pair of terms by circling one of the numbers provided or “DK -- Don’t Know
1. Central Tendency Mean(Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK
2. Hypothesis Description(Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
34
Concept x Concept Proximity Matrix For A Student
Cent. Tend. Corr. Desc. Regr. ANOVA Hyp. Inf. Mean Median Mode Pop. Sample Stat. t-test Variab. Variance
Central Tendency 0Correlation 8 0Description 1 4 0Regression 9 3 8 0ANOVA 9 7 9 1 0Hypothesis 8 7 8 3 3 0Inference 8 8 5 2 3 1 0Mean 1 6 2 9 9 9 8 0Median 1 8 6 8 9 9 8 3 0Mode 1 8 4 9 8 9 8 3 2 0Population 8 8 8 3 3 2 1 8 7 7 0Sample 3 5 2 8 8 6 2 6 5 8 1 0Statistics 1 6 1 4 7 4 1 6 5 6 3 2 0t-test 8 8 7 2 1 4 2 7 8 9 4 9 7 0Variability 7 8 1 9 7 9 9 8 8 9 9 3 4 8 0Variance 8 7 3 8 5 8 7 7 9 8 8 3 6 8 1 0
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
35
Variance
Variability
Description
Mean
Central Tendency
Mode
Median
Statistics
Inference
Hypothesist-test
ANOVARegression
Correlation
Population
Sample
Network Representation of Proximities
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
36
Small Pilot Study (N=28) Of Similarity Judgments And:
• Concept maps
• Multiple choice
Terms: Variability, Variance, Standard Deviation
Variability
VarianceStd. Dev.
measuresmeasures
is the square root of
1. Which of the following refers to a procedure for making generalizations from a limited random sample of data?
a. Statistical Inferenceb. Intuitionc. Population Parameterizationd. Standardization
Source: Ed Wiley’s 2nd Year Project
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
37
Inference
Sample
Hypothesis
Population
ANOVA
LinearRegression
refers to relationships within the
uses sample to test ideas in the
is used
to te
st ideas
about popula
tion i
n
is used to test
hypotheses in
is used to test population-related
address es ide as abou t vari ables in pop ul at io n in the p ro cess of
is a procedure in statistical
is one special case of the general
linear model, along with
is a
pro
cedu
re in
stat
istic
al
Sample Portion Of Concept Map
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
38
Correlation Among Methods
Type of Assessment
PretestConcept Map .155 Similarity Judgment .554* .251
PosttestConcept Map .330Similarity Judgment .434* .706*
Multiple Choice Concept Map
* p < .01
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
39
What If We Could?
• Collect cognitive structure information indirectly by computer
• Immediately show a student a representation of her knowledge structure
• Permit her to modify the structure• And label the lines?
July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind
40
Conclusions• Concept and cognitive maps appear to tap a
different aspect of declarative knowledge than do multiple-choice tests
• Concept and cognitive maps may provide similar representations of structure although this has not been tested directly
• By assessing different aspects of knowledge, we may broaden what gets taught!