pg 1
Design Research atGlassLab March 25th, 2014
G and g for Formative Assessment
Classroom-based Group Activities
Reusable Digital tools
Core Game Experience
pg 14
Cognitive and Non-cognitive Constructs
•GlassLab is exploring the measurement of a variety of cognitive and non-cognitive constructs• Systems Modeling and Thinking
• Argumentation
• Mathematical modeling
• Problem Solving
• Engagement
•Each GBA is required to address a constellation of competences related to CCSS and/or NGSS as well as 21st century skills
Learning Progressions (LPs) and GBA
•LPs provide a framework to connect and organize
•Learning goals
•Game levels and sequencing
•Assessment construct and psychometric models
•Provide a basis for fast cycle and medium cycle within game feedback and outside of game longer cycle feedback
What’s Notable: an integrated game-based assessment development process
pg 21
pg 22
Evidence Centered Game Design (ECgD)
Domain(Analysis and Modeling)
CAF (Student, Evidence, Task Models)
Reporting Goals
ImplementationDelivery(Four Process)
Sp
rints
Pla
y T
est
ing,
Use
r T
est
ing
Concept
Pre-production
Production
Beta
Release
ECgDBecause the claims are about helping kids learn:
o the competency model is expressed as a set of Learning Progressions
o the evidence model is embedded in design docs along with game and learning components that is organized around an overall instructional pattern
o the detailed evidence and task design is embedded in design docs in conjunction with the detailed game and learning designs (MicroDesign, GDD, etc) along with details in core assessment design doc for (scoring, measurement models, etc).
pg 23
Production and Data Collection Cycle
Post-Production
Production ReleasePre-Production
ConceptingGreenlight
Play TestingSmall Slices
1-2 Players at a Time
Mini Try-outsCognitive Interviews
20 Players
AlphaFirst Quantitative, Refine TelemetryUp to 200 Players
BetaTuning
Up to 500 Players
General AccessContinuous Updates
Adding Features
Most Changes to Cognitive
and Scoring Models
Most Changes to Game Play
and Mechanics
Most Changes to Bayes’
Priors
Phase
Studies (N= per
task)
Mid Concept/early
pre-pro
Think-aloud
N = 10
Play-testing
N = 10
Mini-tryouts
N = 50
Large-scale tryoutsN = 200+
Late pre-production
Late Production
Field Test N=1000+
(Beta) Convenience
sample
User Studies: Why, size, outcomes, time frame, data
Develop early hypothesesabout cognition, performance,and behaviors
Examine task performance and early evidence of Behaviors. Confirm hypotheses on cognition. Draft scoring algorithms. Careful sampling
Analyze performance and observable behaviors. Evaluate scoring algorithms.Careful sampling
VerbalizationsItem responses
InteractionsItem responses
InteractionsItem responses
Outcomes
Types of Data
Verbalizations
Gather direct evidence for cognitions underlying performance and interactive behaviors; confirm construct validity
Option: Verbal Protocol
N = 20
Calibrate psychometric models. Refine feedback
Beta
pg 27