pg 1

Design Research atGlassLab March 25th, 2014

G and g for Formative Assessment

Classroom-based Group Activities

Reusable Digital tools

Core Game Experience

pg 14

Cognitive and Non-cognitive Constructs

•GlassLab is exploring the measurement of a variety of cognitive and non-cognitive constructs• Systems Modeling and Thinking

• Argumentation

• Mathematical modeling

• Problem Solving

• Engagement

•Each GBA is required to address a constellation of competences related to CCSS and/or NGSS as well as 21st century skills

Learning Progressions (LPs) and GBA

•LPs provide a framework to connect and organize

•Learning goals

•Game levels and sequencing

•Assessment construct and psychometric models

•Provide a basis for fast cycle and medium cycle within game feedback and outside of game longer cycle feedback

What’s Notable: an integrated game-based assessment development process

pg 21

pg 22

Evidence Centered Game Design (ECgD)

Domain(Analysis and Modeling)

CAF (Student, Evidence, Task Models)

Reporting Goals

ImplementationDelivery(Four Process)

Sp

rints

Pla

y T

est

ing,

Use

r T

est

ing

Concept

Pre-production

Production

Beta

Release

ECgDBecause the claims are about helping kids learn:

o the competency model is expressed as a set of Learning Progressions

o the evidence model is embedded in design docs along with game and learning components that is organized around an overall instructional pattern

o the detailed evidence and task design is embedded in design docs in conjunction with the detailed game and learning designs (MicroDesign, GDD, etc) along with details in core assessment design doc for (scoring, measurement models, etc).

pg 23

Production and Data Collection Cycle

Post-Production

Production ReleasePre-Production

ConceptingGreenlight

Play TestingSmall Slices

1-2 Players at a Time

Mini Try-outsCognitive Interviews

20 Players

AlphaFirst Quantitative, Refine TelemetryUp to 200 Players

BetaTuning

Up to 500 Players

General AccessContinuous Updates

Adding Features

Most Changes to Cognitive

and Scoring Models

Most Changes to Game Play

and Mechanics

Most Changes to Bayes’

Priors

Phase

Studies (N= per

task)

Mid Concept/early

pre-pro

Think-aloud

N = 10

Play-testing

N = 10

Mini-tryouts

N = 50

Large-scale tryoutsN = 200+

Late pre-production

Late Production

Field Test N=1000+

(Beta) Convenience

sample

User Studies: Why, size, outcomes, time frame, data

Develop early hypothesesabout cognition, performance,and behaviors

Examine task performance and early evidence of Behaviors. Confirm hypotheses on cognition. Draft scoring algorithms. Careful sampling

Analyze performance and observable behaviors. Evaluate scoring algorithms.Careful sampling

VerbalizationsItem responses

InteractionsItem responses

InteractionsItem responses

Outcomes

Types of Data

Verbalizations

Gather direct evidence for cognitions underlying performance and interactive behaviors; confirm construct validity

Option: Verbal Protocol

N = 20

Calibrate psychometric models. Refine feedback

Beta

pg 27