Embed Size (px)
DESCRIPTION
Using Concept Maps for Evaluation. Combines results from 2 projects conducted across 2 years: Using Concept Maps Transcribed from Interviews to Quantify the Structure of Preschool Children’s Knowledge About Plants (Wehry, S., Algina, J., Hunter, J., & Monroe-Ossi, H.) & - PowerPoint PPT Presentation
Citation preview
4251
0011 0010 1010 1101 0001 0100 1011
Using Concept Maps for EvaluationUsing Concept Maps for Evaluation
Combines results from 2 projects conducted across 2 Combines results from 2 projects conducted across 2 years:years:
Using Concept Maps Transcribed from Interviews to Using Concept Maps Transcribed from Interviews to Quantify the Structure of Preschool Children’s Quantify the Structure of Preschool Children’s Knowledge About Plants Knowledge About Plants (Wehry, S., Algina, J., Hunter, J., & (Wehry, S., Algina, J., Hunter, J., & Monroe-Ossi, H.)Monroe-Ossi, H.)
&Healthy Habits Through Literacy: A Concept Mapping Healthy Habits Through Literacy: A Concept Mapping
Curriculum for Preschool and Prekindergarten Children Curriculum for Preschool and Prekindergarten Children (Monroe-Ossi, H.,(Monroe-Ossi, H., Wehry, S., Algina, J., & Hunter, J.)Wehry, S., Algina, J., & Hunter, J.)
The Florida Institute of Education at the University of North FloridaThe Florida Institute of Education at the University of North Florida
4251
0011 0010 1010 1101 0001 0100 1011
Using Concept Maps for EvaluationUsing Concept Maps for Evaluation
A shared goal of the A shared goal of the Young Florida NaturalistsYoung Florida Naturalists & & Healthy Habits Through LiteracyHealthy Habits Through Literacy projects – examine projects – examine concept mapping as a tool to quantify conceptual concept mapping as a tool to quantify conceptual development in preschool children.development in preschool children.
Researchers need a reliable and valid way to use Researchers need a reliable and valid way to use children’s concept maps for program evaluation.children’s concept maps for program evaluation.
4251
0011 0010 1010 1101 0001 0100 1011
Scoring System: 3 ComponentsScoring System: 3 Components
Propositions• 0 points if irrelevant0 points if irrelevant• 1 point if an example of concept1 point if an example of concept• 2 points if describes an attribute of concept2 points if describes an attribute of concept• 3 points if states a purpose of the attribute3 points if states a purpose of the attribute
Cross-Links
• 5 points each5 points each
Hierarchical Levels
• 5 points each5 points each
4251
0011 0010 1010 1101 0001 0100 1011
Scoring System: ExamplesScoring System: Examples
4251
0011 0010 1010 1101 0001 0100 1011
Interrater Reliability StudyInterrater Reliability Study
DataData: Young Florida Naturalists : Young Florida Naturalists Assessment 2 Interviews (Assessment 2 Interviews (n n = 48).= 48).
Maps: 1 researcher mapped all interviews.Maps: 1 researcher mapped all interviews. Raters: 3 trained researchers rated all maps. Raters: 3 trained researchers rated all maps. Analysis: Estimated how much variance Analysis: Estimated how much variance
was accounted for by raters, maps, and was accounted for by raters, maps, and raters*maps.raters*maps.
4251
0011 0010 1010 1101 0001 0100 1011
Interrater Reliability Study: ResultsInterrater Reliability Study: Results
CCoonncceepptt MMaapp SSccoorree
PPrrooppoossiittiioonnss
CCrroossss--LLiinnkkss
HHiieerraarrcchhiiccaall LLeevveellss
TToottaall
SSoouurrccee ooff VVaarriiaattiioonn
PPeerrcceenntt ooff TToottaall
VVaarriiaabbiilliittyy
PPeerrcceenntt ooff TToottaall
VVaarriiaabbiilliittyy
PPeerrcceenntt ooff TToottaall
VVaarriiaabbiilliittyy
PPeerrcceenntt ooff TToottaall
VVaarriiaabbiilliittyy Maps (m) 86% 71% 83% 95% Raters (r) 2% 5% 1% 1% m*r 12% 25% 16% 5% Reliability .95 .90 .94 .98
4251
0011 0010 1010 1101 0001 0100 1011
Concept Mapping ProtocolConcept Mapping Protocol
The 3 Young Florida Naturalist raters used the 48 interviews to develop a mapping protocol.• Propositions do not have to be grammatically Propositions do not have to be grammatically
correct.correct.• Do not map irrelevant propositions.Do not map irrelevant propositions.• Mapped propositions can stand alone, that is, not Mapped propositions can stand alone, that is, not
connected to the focal concept.connected to the focal concept.• When in doubt, the benefit goes to the child.When in doubt, the benefit goes to the child.• Score the resulting concept maps to test ease of Score the resulting concept maps to test ease of
scoring.scoring.
4251
0011 0010 1010 1101 0001 0100 1011
Interrater & Intermapper Reliability Interrater & Intermapper Reliability StudyStudy
DataData: Healthy Habits Through Literacy : Healthy Habits Through Literacy Assessment 2 Interviews (Assessment 2 Interviews (n n = 48).= 48).
Maps: 3 researchers mapped all interviews.Maps: 3 researchers mapped all interviews. Raters: 3 trained researchers rated all maps. Raters: 3 trained researchers rated all maps. Analysis: Estimated how much variance was Analysis: Estimated how much variance was
accounted for by the interviews, mappers, accounted for by the interviews, mappers, raters, and all interactions of the 3 sources.raters, and all interactions of the 3 sources.
4251
0011 0010 1010 1101 0001 0100 1011
Interrater & Intermapper Reliability: Interrater & Intermapper Reliability: ResultsResults
CCoonncceepptt MMaapp SSccoorree
PPrrooppoossiittiioonnss
CCrroossss--LLiinnkkss
HHiieerraarrcchhiiccaall LLeevveellss
TToottaall
SSoouurrccee ooff VVaarriiaattiioonn
%% ooff TToottaall VVaarriiaabbiilliittyy
%% ooff TToottaall VVaarriiaabbiilliittyy
%%ooff TToottaall VVaarriiaabbiilliittyy
%% ooff TToottaall VVaarriiaabbiilliittyy
Interview (i) 96% 86% 86% 95% i*m 3% 12% 10% 0% i*r 1% 2% 4% 0% m*r 1% 0% 0% 0% i*m*r 0% 1% 3% 5% Reliability .96 .85 .83 .95
4251
0011 0010 1010 1101 0001 0100 1011
ValidityValidity• Young Florida Naturalist - Prekindergarten children
were assessed in the fall and spring (n = 26).
• The Test of Early Reading Ability -Third EditionThe Test of Early Reading Ability -Third Edition (TERA-3)(TERA-3)
• Bracken Basic Concept Scale -Third Edition: Bracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R)Receptive (BBCS-3:R)
• Average Concept Map ScoresAverage Concept Map Scores
4251
0011 0010 1010 1101 0001 0100 1011
Validity Study: ResultsValidity Study: ResultsCorrelations with Concept Map ScoresCorrelations with Concept Map Scores
FFaallll SSpprriinngg TTeesstt SSuubbtteesstt rr pp rr pp
TERA- Alphabet .14 .49 .42 .03** 3 Conventions .08 .71 .27 .19 Meaning .39 .04** .21 .30 BBCS- SRC .12 .56 .36 .07* 3:R Direction/Position .13 .52 .30 .14 Self-/Social Aware .19 .36 .40 .05** Texture/Material .01 .97 .36 .08* Quantity .25 .21 .17 .40 Time/Sequence .05 .81 .12 .56
Note: *p < .10; ** p < .05; SRC is the School Readiness Composite.
4251
0011 0010 1010 1101 0001 0100 1011
Validity: Regression ResultsValidity: Regression Results
• After controlling for gender, age, prekindergarten class, and pretest score, the children’s concept map scores predicted spring:• BBSC - 3:R Texture/Material scale BBSC - 3:R Texture/Material scale
achievement.achievement.• BBSC - 3:R Self-/Social Awareness scale BBSC - 3:R Self-/Social Awareness scale
achievement.achievement.
4251
0011 0010 1010 1101 0001 0100 1011
Validity 2: MeasuresValidity 2: Measures
• Healthy Habits Through Literature -Prekindergarten and preschool children were assessed in the spring (n = 35) using the following measures:
• The Peabody Picture Vocabulary Test - Fourth EditionThe Peabody Picture Vocabulary Test - Fourth Edition (PPVT-4).(PPVT-4).• The Expressive Vocabulary Test – Second EditionThe Expressive Vocabulary Test – Second Edition (EVT-2).(EVT-2).• Bracken Basic Concept Scale -Third Edition: ReceptiveBracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R).(BBCS-3:R).• Bracken Basic Concept Scale -Third Edition: ExpressiveBracken Basic Concept Scale -Third Edition: Expressive (BBCS-3:E).(BBCS-3:E).• Average Concept Map Scores.Average Concept Map Scores.
4251
0011 0010 1010 1101 0001 0100 1011
Validity 2: ResultsValidity 2: Results Correlations with Concept Map ScoresCorrelations with Concept Map Scores Test Subtest r p PPVT-4 .19 .26 EVT-2 .29 .09* BBSC-3:R SRC .19 .27 Self-/Social Aware .49 .00** Texture/Material .25 .14 BBSC-3:E SRC .20 .24 Self-/Social Aware .23 .18 Texture/Material .23 .18
Note: *p < .10; ** p < .05; SRC is the School Readiness Composite
4251
0011 0010 1010 1101 0001 0100 1011
ConclusionsConclusions
Concept maps developed from preschool Concept maps developed from preschool children’s transcribed interviews can be children’s transcribed interviews can be reliably scored.reliably scored.
Preschool children’s transcribed interviews Preschool children’s transcribed interviews can be reliably mapped and scored.can be reliably mapped and scored.
Resulting concept map scores correlated with Resulting concept map scores correlated with measures of expressive vocabulary, attributes measures of expressive vocabulary, attributes of people & objects, and alphabet knowledge.of people & objects, and alphabet knowledge.
4251
0011 0010 1010 1101 0001 0100 1011
DiscussionDiscussion
Revise Scoring System• 1. To reflect the use of 1. To reflect the use of DynamicDynamic vs. Staticvs. Static Statements.Statements.• 2. Concerning the use of examples.2. Concerning the use of examples.
Revise Mapping ProtocolRevise Mapping Protocol• 1. To reduce variance due to mapper in cross-1. To reduce variance due to mapper in cross- link & hierarchical level scores.link & hierarchical level scores.• 2. To better fit across curricula.2. To better fit across curricula.
4251
0011 0010 1010 1101 0001 0100 1011
DiscussionDiscussion
• Why rate concept maps rather the interviews?• Rating the interview did not provide clarity in Rating the interview did not provide clarity in
determining the structure of children’s knowledge determining the structure of children’s knowledge relative to the domain (Novak & Musondo, 1991) relative to the domain (Novak & Musondo, 1991)
• Concept mapping children’s interviews makes Concept mapping children’s interviews makes visible the their propositions, cross-links, and visible the their propositions, cross-links, and hierarchical structure.hierarchical structure.
• The concept maps make visible places where The concept maps make visible places where instruction can be improved.instruction can be improved.
4251
0011 0010 1010 1101 0001 0100 1011
Next StepsNext Steps
• Professional Development:• 22 prekindergarten teachers to use concept maps to 22 prekindergarten teachers to use concept maps to
facilitate children’s learning.facilitate children’s learning.• Kindergarten teachers to use concept maps to help Kindergarten teachers to use concept maps to help
children connect their background knowledge to children connect their background knowledge to newly acquired information.newly acquired information.
• Middle School Implementation:• Use concept mapping strategies with 6Use concept mapping strategies with 6thth and 7 and 7thth
graders in after-school programs at two middle graders in after-school programs at two middle schools. schools.
