Evidence-Centered Game Design Kristen DiCerbo, Ph.D. Principal Research Scientist, Pearson Learning Games Scientist, GlassLab

Evidence-Centered Game Design

Kristen DiCerbo, Ph.D.Principal Research Scientist, PearsonLearning Games Scientist, GlassLab

Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved. 2

The ability to capture data from everyday events should fundamentally change how we think about learning and assessment.


Evidence-Centered Assessment Design

• What complex of knowledge, skills, or other attributes should be assessed?

• What behaviors or performances should reveal those constructs?

• What tasks or situations should elicit those behaviors?

Student Model

Evidence Model(s)

Measurement Model

Scoring Model

X1

Task Model(s)

1 . x x x x x x x x 2 . x x x x x x x x

3 . x x x x x x x x 4 . x x x x x x x x 5 . x x x x x x x x 6 . x x x x x x x x 7 . x x x x x x x x 8 . x x x x x x x x

X2

X1

X2

Mislevy, Steinberg, & Almond (2003)

Two Examples


Student Model - Aspire


Default Gateway

Proficiency

Understand the Purpose

ObtainNavigate Devices

Local vs. RemoteTest

Configuration

Basic IP Addressing

From Shute & Ventura, 2012

Task Model - Aspire


Evidence Model - Aspire


Student Model – Mars Generation One


Level Interpretation Expression

Achievement AchievementPreliminary Reason-1-I-A

Identifies reasons people give to support a specific point

Reason-1-E-AGenerates at least one reason to support a specific point, in sentence form

Foundational Reason-2-I-AIdentifies supporting reasons or evidence in an argument and relates them to the point they support

Reason-2-E-AGenerates multiple reasons to support a point, and uses these reasons to counter others' argument in an engaging, familiar context

Basic Reason-3-I-ARecognizes and explains the relationship between main and supporting points and keeps track of which evidence supports which point

Reason--3-E-ABuilds logical, hierarchically structured arguments by selecting and arranging reasons and evidence to support main and subsidiary points

Intermediate Reason-4-I-AIdentifies specific points in a text that are vulnerable to objections and counterarguments

Reason-4-E-ACreates simple critiques or rebuttals that critically provide summaries of or responses to other people’s arguments


Task Model – Mars Generation One

Game mechanics

• Matching game mechanics and learning mechanics• “The nature of the construct being assessed should guide

the selection of relevant task…” Sam Messick, 1994• A game is built around a core mechanic• A mechanic is a set of verbs that create a core loop• If the core doesn’t match the competency, the competency

isn’t integrated.


Evaluate evidence

Construct argument

Critique argument


Evidence – Mars Generation One


skill milestones / tasks standards (CCSS) game phases/actions

Evidence action #1 (specific actions that provide evidence)

identify 1 piece of data that relates to specific argument

CCSS.ELA-Literacy.WHST.6-8.1a, CCSS.ELA-Literacy.W.6.1a keeping, sorting, discarding

constructing dual-core claim/data pair for argubots (google doc: Loop 1 Processes, Stage 1.3)

identify >1 piece of data that relates to a specific argument

CCSS.ELA-Literacy.WHST.6-8.1a; CCSS.ELA-Literacy.WHST.6-8.1b; CCSS.ELA-Literacy.W.6.1b

sorting, organizing pre-battle

constructing dual-core claim/data pair for argubots

identify data as pro/con

CCSS.ELA-Literacy.WHST.6-8.1a, CCSS.ELA-Literacy.W.6.1a

sorting, discarding, (not) using in battle

accuracy of using the relevancy switch/toggle during dual-core construction

identify data as valid/invalid

CCSS.ELA-Literacy.RH.6-8.8 ; CCSS.ELA-Literacy.RST.6-8.8

sorting, discarding, (not) using in battle

relevancy switch/toggle during dual-core construction

identify common arugmentation schemes

CCSS.ELA-Literacy.RH.6-8.8 ; CCSS.ELA-Literacy.RST.6-8.8

sorting, interaction w NPC/computer, ogranizing pre-battle, battle

appropriateness of argubot selected for dual-core

Identification of Argument elements

rebuttalMultiplerebuttalSinglerelationsroles

25.025.025.025.0

Organize elements of an argument

MultipleBackingMultipleSchemesMultipleData

33.333.333.3

Use of Argument Elements

CritiqueRebuttalMultipleDataSchemesDatabelowData

25.025.025.025.0

Evaluate Argument Elements

ComplexArgumentCantEvaluateSimpleArgument

33.333.333.3

Main Argumentation Competency based u...

PreliminaryFoundationalBasicIntermediate

30.040.020.010.0


Does it really get at what we think it does?

Cognitive Processes


0:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 24:00 26:00

Pair 11

Pair 10

Pair 9

Pair 8

Pair 7

Pair 5

Pair 3

Pair 6

Pair 4

Pair 2

Pair 1

Problem Identification

Solution Generation/ Implementation

Solution Evaluation

DiCerbo, Frezzo, & Deng, 2011


Thank You

[email protected]

References

• DiCerbo, K. E. & Behrens, J. T. (2014). Impact of the digital ocean on education. London: Pearson. Available: http://research.pearson.com/digitalocean

• DiCerbo, K. E., Frezzo, D. C., & Deng, T. (2011). Substantive validity of a simulation-based game. Research and Practice in Technology-Enabled Learning, 6(3), 161-185. Available: http://apsce.net/RPTEL/RPTEL2011NovIssue-Article2_pp161-185.pdf

• Shute, V. J. & Ventura, M. (2013). Stealth assessment in digital games. White paper for MIT series, published by the MacArthur Foundation. Available: http://myweb.fsu.edu/vshute/pdf/white.pdf

For the masochist:Mislevy et al. (2014). Psychometric Considerations in Game-Based

Assessment. http://www.instituteofplay.org/work/projects/glasslab-research/


Documents

Evidence-Centered Game Design Kristen DiCerbo, Ph.D. Principal Research Scientist, Pearson Learning Games Scientist, GlassLab