Upload
ursula-bates
View
213
Download
0
Embed Size (px)
Citation preview
Assessment and Instruction
Tests often favor full-blown declarative and procedural knowledge. They miss the value of many other types of learning,
and pull for the “mastery” of facts and steps. Preparation-for-future-learning tests can lift out
value of non-paradigmatic learning experiences. Tests include opportunities to learn, and we measure
whether prior learning experiences prepare people to learn from those opportunities.
Experience and Explanation
Hidden value of videogames.Dylan ArenaKidaptive.Com
Arena, D. A., & Schwartz, D. L. (2013). Experience and explanation: Using videogames to prepare students for formal instruction in statistics. Journal of Science Education and Technology.
Interactive PFL
A limitation of PFL research has been the static nature of the learning resources in the PFL tests. Lecture, worked example, text.
A good deal of learning comes through interaction. Self-directed, interactive learning involves a series
of choices. Students who are prepared for future learning
should make good choices about learning.
Choice as the proper outcome of education.
• No time for a deeper discussion about normative goals of education and appropriateness of measuring choice.
• But see over here
• Here, we just present an extended example for consideration.
Free download at MIT Press
1) Choose Booth 2) Design Poster 3) Choose Focus Group
4) Choose - or + Feedback
TestPoster
5) Read Feedback
7) Post Poster(see Ticket Sales)
Revise?
No
Yes
6) Re-design Poster
Posterlet DorisChin
KristenBlair
MariaCutumisu
Feedback
Both positive (“I like…”) and negative (“I don’t like…) feedback are informative in the game. Built a system that evaluates 21 graphic design elements. Negative does not mean punishing.
Literature indicates that negative feedback better for learning, but risk of ego threat.
To my knowledge, nobody has examined the choice to seek negative feedback. Central to many “design thinking” pedagogies.
Research design
Managed to get about 450 children at two different “game-themed” schools Quest schools in NYC and Chicago.
Children played posterlet for about 20 minutes. We tracked:
Number of times they chose negative feedback (0-9) Number of times they revised (0-3) Quality of posters (based on 21 features) Performance on a posttest of 21 graphic design principles.
Step 1: Internal Validation
For 2+2, there is no argument that 4 is the correct answer.
With choices, the story is different. People can rightly argue, Who are you to say that choosing negative feedback is the right choice for learning.
To address, we examine correlations between choices and measures of learning or performance.
Evidence on Internal Validation(choosing negative feedback leads to better learning)
Negative Feedback
Revision Poster Quality
Post-test
Negative Feedback
____ .48*** .28*** .23***
Revision ____ .34*** .24***
Poster Quality
____ .39***
13
No differences on 1st poster, so not due to some sort of prior knowledgeof graphic design principles.
Step 2: External Validation
Does behavior in the game world say anything about “real” world learning?
Posterlet finding may not reach outside games. Motivations for various choices in a game may not
reflect motivations in the world.
External Validation(We received standardized test scores for a subset
of the students.)
15
NY (n = 119) Chicago (n = 65)
Reading (ELA)
Math (NY Math)
Reading (ISAT)
Math (ISAT)
Negative Feedback
.33*** .39*** .41*** .33**
Revision.08 .28** .31* .21
Nice to have a replication! It is missing in many data mining efforts, not to mention the very idea of external validation.
Step 3: Experience Validation
Ultimately, we want to measure experiences not individuals. Want to help improve instruction and flag kids who could
use some attention.
To know if we can measure the effects of experience, we need to know if experience has an influence on the assessment.
Summary
Choice to seek negative feedback correlated with better learning. Is there another demonstration like this out there?
Seeking negative feedback seems to be a more stable correlate of learning than choosing to revise.
Like all assessments, if you do not have control over the experience, it is hard to know what experience you are measuring.
Current Activities
Now that we can measure learning choices (so far), we want to show that it is possible to create experiences that influence choice behaviors.
Working to build brief design thinking curricula. Complementary suite of design thinking choicelets. Conduct an experiment where some get curricula, some not.
We’ll see. In the meantime…
Emerging Design Principles PFL Principle
Must be something new to learn within the assessment. Choice Principles
Learning must depend on making a choice. Game can be completed without choosing to learn. (It is an unforced choice.)
Typical Performance Principle Game should be designed to measure typical behavior not test behavior.
Validation Principles Internal – show the choice does lead to learning in game External – show the choice correlates with learning outside game. Experimental – show experiences outside game can influence choices.
Data Abstraction Principles Conceptualize game as having different “regions” of choice. Aggregate regions of choices within rounds, and analyze rounds sequentially.