ATLT Final Technical Report 2012-12-31 v1

8/9/2019 ATLT Final Technical Report 2012-12-31 v1

1/88

Title: Game-Based Tutoring Technologies Final Report

Date: December 31, 2012

repared under: ! TR"#$# %ontract & !'()*+-12-%-)0))

repared b:

"danced Training . /earning Technologies, //%

+++* %orporation /ane, #uite 20(irginia Beach, irginia 23+'2

Technical %:

Daid B %aitt, hD, %!#

dbcaittatlt-llccom

4(*(5 *'0-0033

"pproed b:

rgani6ation 7ame #ignature Date

Document %ontrol 8n9ormation

Reision Reision istor Date

Ver. 1.0 Final Technical Report November 30, 2012

Ver. 1.1 Clarifications/responses to Government !estions. "ecember 31, 2012


2/88

;.

>tili6ation and Dissemination o9 FindingsThe results presented in this Final Technical Report should be of interest toeducational game designers and developers, and personnel involved in theimplementation of intelligent and adaptive tutoring technologies. Thelessons learned and recommended areas for future intelligent tutoringtechnology development should be considered for future R&D activity. Thisreport has been sent to the Government sponsors of this research at PM


3/88

TR!"! and dvanced Distributed #earning $o%#ab. The results are to briefedin December '(.


4/88

Table o9 %ontents

1. Introduction............................................................................1

1.1. Scope of Research & Development................................................1

1.2. Research Objectives.....................................................................2

1.3. Document Overview.....................................................................2

(.).(. *T !oft+are #ibrary *!#-.......................................................................... (.).. *TMath Game /valuations......................................................................

2. Intellient !utorin "ent Software #ibrar$ %IS#.......................3

2.1. IS# and I!" "rchitecture Overview................................................3

.(.(. *!# $omputer !oft+are $on0guration *tems.............................................).(.. The *!# Development #ife $ycle................................................................)

2.2. I!" 'eatures & (apabilities...........................................................)

..(. *T *nstructional !e1uencing.....................................................................2... !tructure of 3no+ledge.............................................................................4..). ctive Tutoring lgorithms......................................................................('..2. *T /nhancements 5 Phase ) /valuation !upport....................................()

2.3. !he I!" "rchitecture...................................................................1*

.).(. The *T $oncept Model............................................................................(6

.).. The *T rchitecture................................................................................(6.).). The *T 7b8ect Model..............................................................................(42.). I!" "pplication +rorammer,s Interface %"+I..............................1-

.2.(. *T *nitiali9ation....................................................................................... '.2.. ssessment $reation...............................................................................(.2.). ssessment............................................................................................. .2.2. Recommendation Process....................................................................... )

2.*. IS# #essons #earned...................................................................2)

2.. 'uture Directions.......................................................................2*

3. I!"/0ath ame valuations....................................................2

3.1. Introduction...............................................................................2

3.2. +urpose.....................................................................................2

3.3. 0ethodolo$ and Research 4uestions.........................................2

).).(. Phase 7ne Methodology..........................................................................:


5/88

).).. Phase T+o Methodology..........................................................................:).).). Phase Three Methodology.......................................................................;

3.). Results......................................................................................35

).2.(. Phase 7ne Results...................................................................................)').2.. Phase T+o Results................................................................................... )

).2.). Phase Three Results................................................................................)23.*. DoD" "ctivit$...........................................................................)1

3.. Summar$ of 'indins..................................................................)1

3.. 'uture Directions.......................................................................)1

3.6. "dditional #essons #earned........................................................)2

). References............................................................................)3

).1. "cron$ms...................................................................................)3

).2. References.................................................................................)3

"ppendi7es................................................................................))

"ppendi7 1.......................'ormative valuation +hase One +articipant4uestionnaire.............................................................................))

"ppendi7 2.......................'ormative valuation +hase !wo +articipant4uestionnaire.............................................................................)-

"ppendi7 3.. +hase !hree 'ormative valuation 8 (umulative +re8!est

*-

"ppendi7 )...............+hase !hree 'ormative valuation 8 0ini +re8!est)

"ppendi7 *.+hase !hree 'ormative valuation 8 (umulative +ost8!est3

"ppendi7 ..........+hase !hree 'ormative valuation 8 !eacher Surve$6

"ppendi7 ..........+hase !hree 'ormative valuation 8 Student Surve$65

"ppendi7 6.......+hase !hree 'ormative valuation 8 9loom,s 0appin61


6/88

/ist o9 Figures

Fi%!re 1. -T' -nstr!ctional (e!encin% ? 57ample.........................................................................@

Fi%!re 2. -T' "ata :o#el + (tr!ct!re of Ano&le#%e......................................................................;

Fi%!re 3. -T' i%h evel Class "ia%ram......................................................................................1;

Fi%!re B. -T'5n%ine:ana%er Class -nterface...............................................................................1

Fi%!re @. 'ssessment>!il#er -nterface..........................................................................................1

Fi%!re ;. 'ssessment -nterface......................................................................................................1

Fi%!re . 'ssetRecommen#ation5n%ine Relate# Classes.............................................................1loom8s.


17/88

@. ($nthesis ):ae it Ne&* + The s$nthesis level of thinin% ass the st!#ent to pla$ aro!n#

&ith ne& information an# form ne& ima%es. The no&le#%e the st!#ent receives is

combine# &ith &hat the st!#ent alrea#$ has to mae a ne& connection. (ome process&or#s for s$nthesis are ima%ine, combine, role+pla$, compose, invent, pre#ict, create,

#esi%n, a#apt, #evelop.

;. 5val!ation )Q!#%e -t* + The eval!ation level of thinin% ass a st!#ent to =!#%e accor#in%to some stan#ar#. ' !estion can as the st!#ent to i#entif$ an important criterion to

complete a tas, or as for the st!#ent to rate somethin% base# !pon a pre#etermine#

criterion.

'ppen#i7 < provi#es the co#in% of assessment items to ill!strate the relationship bet&een

problems/problem %ro!ps an# >loomJs Ta7onom$. The co#in% metho# &as #evelope# an#e7pan#e# b$ Qohn :a$nar#.

2.2.3. "ctive !utorin "lorithms

This section #isc!sses the e$ t!torin% al%orithms !se# to activel$ monitor an# mo#if$ a

st!#ent8s instr!ctional environment. 's &as previo!sl$ #one, &e #escribe the -T' feat!res an#al%orithms as !se# in the conte7t of the :ath Game. The follo&in% feat!res an# al%orithms &ere

a part of the initial e7traction of the -T' from the :ath Game. From the perspective of t!torin%

feat!res, the e$ al%orithms can be s!mmari9e# as follo&sE

Table 1 8T" %apabilities Features

8T" Feature Description

Generativit$ Fee#bac is %iven #!rin% practices problems )hint* &hile !sin% the tablet an# afterassessment )score*

(t!#ent :o#elin% "omain mo#els for e7pert an# act!al st!#ent no&le#%e challen%e cate%ori9ationfor hi%h+or#er thinin% assessments

57pert :o#elin% (ill threshol#s for st!#ent performance on assessment problems.

-nteractive earnin% :ath challen%es an# %ame pla$ for st!#ent en%a%ement appropriatel$ conte7t!ali9e#an# #omain+relevant )e.%., the st!#ent, thro!%h his/her avatar is re!ire# to solve

math problems to constr!ct a to&n festival as part of the %ame stor$*

-nstr!ctional :o#elin% Re!ire# an# relevant sill content )e.%., in the conte7t of the math e#!cational

%ame, this is the c!rric!l!m math topics


18/88

The al%orithms are or%ani9e# into t&o primar$ f!nctional areasE 1* monitorin%, an# 2*

instr!ctional intervention, an# are s!mmari9e# belo&. "etails on the al%orithm are presente# in

the s!bsections belo&.

1. :onitorin% hase

a. '%%re%ate st!#ent performance for all in#ivi#!al sills &ithin the c!rrentassessment instance for a problem %ro!p or challen%e in the %ame.

b. Comp!te best possible score per sill

c. -f an$ best possible score falls belo& the specifie# minim!m acceptable scorethen tri%%er intervention

2. -nstr!ctional -ntervention hase

a. Collect sills in#ices that fall belo& the acceptable minim!m threshol# for c!rrentassessment instance

b. resent st!#ent &ith a #ashboar# that contains instr!ctional assets specificall$

tar%ete# to&ar# the #eficient sills. 's these assets are !se#, information abo!ttimes accesse# are store#.

2.2.3.1 enerativit$ ? Instructional 'eedbac>

-n the :ath Game each practice problem is broen #o&n into in#ivi#!al steps. The -T'

monitors the st!#ent8s response for each step in the problem solvin% se!ence. -f an$ step isans&ere# incorrectl$ a hint is reveale#. (ome items have m!ltiple hints associate# &ith a

partic!lar s!b+step. For e7ample if a !estion re!ires a st!#ent to inp!t a fraction there &ill be

t&o in#ivi#!al te7tbo7es, one for the n!merator an# one for the #enominator. The s$stem ma$%ive a separate hint if the n!merator is correct b!t the #enominator is incorrect, than if the

opposite occ!rs.

The hints themselves are static in nat!re, b!t are reveale# in a manner that is sensitive to

&hichever specific inp!t &as incorrect. o&ever the fee#bac #oes not var$ base# on &hat &as

entere#, onl$ &here in the se!ence of steps the incorrect response &as entere#. The t!tor alsoprovi#e positive fee#bac !pon s!ccessf!ll$ completion of problems.


19/88

2.2.3.2 enerativit$ ? "ssessment 'eedbac>

'n assessment is comprise# of !estions that ma$ cover m!ltiple sills. ' separate score iscalc!late# for the completion of that specific sill as &ell as the entire assessment. This allo&s

the s$stem to #ifferentiate the fee#bac for the pla$er at the en# of the assessment that &ill sho&

the in#ivi#!al scores for each sill attempte# #!rin% an$ challen%e.

'cc!m!lator overall'cc!m!lator'cc!m!latorS acc!m!lators

foreach)L!estion in 'ssessment*

if)+UisCorrect)*

acc!m!lators+Usill-n#e7S.incrementCorrect)*overall'cc!m!lator.incrementCorrect)*

else

acc!m!lators+Usill-n#e7S.increment-ncorrect)*

overall'cc!m!lator.increment-ncorrect)*

println)WHverall (coreE W K overall'cc!m!lator.%etercenta%e)* *

foreach)'cc!m!lator a in acc!m!lators* println) a.%et(illName)* K W(coreE WK a.%etercenta%e)* *

2.2.3.3 !eacher/Instructional 0odelin 8 Di@erentiated instruction

hile the assessment is on%oin% each sill %ro!p is monitore# to force an interr!ption once the

st!#ent is no lon%er capable of achievin% a passin% score on the assessment. These tri%%ers are%oin% to be fire# at #ifferent times for #ifferent st!#ents base# on their personal performance.

The tri%%er e7poses the specific sill the st!#ent &as havin% tro!ble &ith, an# allo&s for the

instr!ction to be tailore# to that sill. This is tr!e for both assessment problems an# challen%eproblems.

foreach)L!estion in 'ssessment* if)+UisCorrect)*

acc!m!lators+Usill-n#e7S.incrementCorrect)*

overall'cc!m!lator.incrementCorrect)* else

acc!m!lators+Usill-n#e7S.increment-ncorrect)*

overall'cc!m!lator.increment-ncorrect)*

if)acc!m!lators+Usill-n#e7S.%etTheoretical>est(core)* UX

lo&est'cceptable(core* trigger8nterruption4=-Csill8nde


20/88

2.2.3.) enerativit$ and Interactive #earnin ? (ontent+resentation

The -T' monitors an# presents the specific sill content nee#e# to the st!#ent )list of assetsnee#e# for the sill*. hile the content list is static, it is possible to a#=!st fee#bac to the

st!#ent base# on the monitorin% res!lts. hen an interr!ption is processe#, assets are selecte#

that correspon# to the sill that tri%%ere# the interr!ption. The st!#ent is informe# specificall$an# #eclarativel$ &hich sill the$ are havin% tro!ble, then the$ are presente# &ith a list of assetsthat are specificall$ selecte# beca!se the$ a##ress that specific sill an#/or st!#ent #eficienc$.

2.2.3.* 7pert/Instructional 0odelin ? Remediation andinstructional strateies

The -T' maintains a set of threshol#s that have been #efine# for ever$ sill an# the performance

is monitore# for those sills #!rin% assessments. hen the threshol#s are crosse# the -T'initiates a response that re#irects the pla$ers into a reme#iation process &here the$ are sho&n

some instr!ctional assets that instr!ct the #eficient sill. The instr!ctional assets have been

#esi%ne# b$ (!b=ect :atter 57perts )(:5s* !tili9in% instr!ctional best practices an# metho#sthat are appropriate to the sills an# concepts bein% presente#. The acceptable performance

threshol# #efines &hat it &o!l# mean to an e7pert mo#el. (t!#ents are onl$ allo&e# to pro%ress

&hile the$ sta$ &ithin the bo!n#s of this mo#el. Hnce the performance crosses o!tsi#e of thismo#el a corrective action is initiate#.

2.2.). I!" nhancements ? +hase 3 valuation Support

This section #oc!ments the si7 -T' soft&are chan%es ma#e to enhance the -T' capabilities.

These mo#ifications &ere incorporate# as a res!lt of the lessons learne# from the hase 1 an#

hase 2 formative eval!ations #isc!sse# in (ection 3 of this report, an# also from collaborative#isc!ssions &ith Government sponsors an# pro=ect staehol#ers.

The first three enhancements specificall$ foc!se# on component interfaces amon% the

t!tor/teacher mo#elin%, instr!ctional mo#elin%, an# st!#ent mo#elin%. These three enhancements

improve# the in+%ame -T' feat!res b$E 1* impactin% the st$le of information presentation )e.%.,

shapes, color, orientation* to help the st!#ent maintain a&areness on the correctness of theirans&ers/responses, 2* allo&in% the st!#ent to #iscern the t$pe/mo#e of information available to

them, an# 3* provi#e the st!#ent &ith a r!nnin% histor$ on their !se of specific so!rces/t$pes of

information.

The remainin% t&o soft&are chan%es are also relate# to st!#ent mo#elin%, t!tor/teachermo#elin%, an# instr!ctional mo#elin% b$ specificall$ capt!rin% the st!#ent e7perience #!rin%

instr!ction. The t&o al%orithms &ere #evelope# to provi#e the -T' &ith an !n#erstan#in% ofst!#ent preferences an#/or aversions for the !se of specific t$pes of learnin% content. The

al%orithm or%ani9es the preferential assets or !n!se# assets into sorte# lists. The componentinterfaces for the %ame are then !se# to #ispla$ the lists to motivate or incite the st!#ent to tr$

proven an#/or alternative content #eliver$ mechanisms to improve learnin% o!tcomes. -n some

cases, the -T' ma$ #eci#e to s!%%est the st!#ent tr$ looin% at relevant an# alternative contentt$pes if the$ #on8t seem to be havin% s!ccess &ith the c!rrent set of content materials the$ have

!se#. -n other cases, the appropriate instr!ctional techni!es ma$ be to enco!ra%e the st!#ent to


21/88

learn !sin% specific t$pes of content materials that have proven to be s!ccessf!l for them in the

past. The follo&in% five s!bsections provi#e a##itional #etail on the soft&are chan%es =!st

#isc!sse#.

2.2.).1 Real8time 'eedbac> durin assessment

The -T' &ill tri%%er an in+%ame vis!al in#icator to the st!#ent to in#icate s!ccess or fail!re forthe c!rrent assessment an#/or challen%e problem. -n the conte7t of the :ath Game &hen apla$er ans&ers a math challen%e !estion a brief vis!al in#icator is #ispla$e# on the screen. -f

the ans&er &as correct a chec is #ispla$e# or if incorrect an I is #ispla$e#. The in#icator

remains on the screen for some #efa!lt time an# is then follo&e# imme#iatel$ b$ the ne7t

problem in the math challen%e !p !ntil an# incl!#in% for the last problem in the challen%e.

2.2.).2 Aisuall$ identif$ asset t$pes

The -T' tri%%ers an in+%ame vis!al in#icator to the st!#ent to alert them to certain t$pes of

learnin% content that is available for their !se. -n the conte7t of the :ath Game &hen a st!#entopens the tablet an# observes &hat t$pes/mo#es of content are available for their !se the are

provi#e vis!al cl!e for !sef!l learnin% assets. 5ach asset t$pe is %iven a #istinct icon that #enotesif the asset is a vi#eo, te7t, or practice problem.

2.2.).3 Aisuall$ mar> used assets

The -T' tri%%ers in+%ame vis!al in#icators to raise st!#ent a&areness abo!t &hat assets that have

been previo!sl$ !se# as &ell as revealin% those assets that have not been previo!sl$ accesse#.

2.2.).) Student !imers

The -T' initiates a series of timers that are !se# to monitor specific activities an# levels ofst!#ent interaction &ith the %ame. The c!rrent set of -T' timers are place# on assessments.

hen the -T' he!ristics loos thro!%h the problems, it checs for times that are belo& threshol#times establishe# for certain t$pes of problems an# sets a fla% to tri%%er an in+%ame response.C!rrentl$, the :ath Game is not respon#in% to the -T' timer. F!t!re implementation ma$ for

e7ample be !se# to #rive responses to a st!#ent &ho is failin% !estions an# also ans&erin%

!estions !icer than e7pecte# for the t$pes of problems the$ are failin% )aa. e$+mashin%behavior*.

2.2.).* Sort "sset #ist b$ Bnused "ssets

Hptimi9e the ser"ataCollection s!ch that it can ret!rn the times an asset &as accesse#. The

overall effect &ill be that the asset list &ill have Yne&Y assets to&ar# the top of the an#previo!sl$ vie&e# assets to&ar# the bottom of the list.

foreach)'sset a in 'ssetCollection*

int times'ccesse# X ser"ata.%etTimes'sset'ccesse#)a*

aXUrimar$(ortAe$ X time'ccesse#

'ssets EX (55CT FRH: 'ssetCollection HR"5R > aXUrimar$(ortAe$ "5(C


22/88

2.2.). Sort "sset #ist b$ Bser +reference

Create an in#e7 on the ser"ataCollection s!ch that asset access or#er can be seen for in#ivi#!allessons. The overall effect &ill be that the asset list &ill have the assets t$pes )i.e. vi#eo, te7t,

practice* that the pla$er accesses first )historicall$ most often* &ill be move# to&ar# the top of

the asset list. This sortin% onl$ starts tain% affect after the st!#ent has faile# an assessment or

challen%e an# has selecte# assets from the tablet. Hn s!bse!ent fail!res, the assets &ill besorte#.

(tren%thVal!e vi#eo(tren%th

(tren%thVal!e te7t(tren%th

(tren%thVal!e practice(tren%th

//-terates over all past lessonsforeach)esson'ccess(e!enceRecor# r in ser"ataCollection*

//Create an #istinct or#ere# set of assets, base# on access or#er

(etZ'ssetT$peU s X r.comp!teHr#ere#(etHf'ssetT$pes)*

//-ncrement the stren%th b$ the access or#er position vi#eo(tren%th KX s.%etosition)vi#eo*

te7t(tren%th KX s.%etosition)te7t*

practice(tren%th KX s.%etosition)practice*

la$er'ssetreference X (ort) vi#eo(tren%th, te7t(tren%th, practice(tren%th S*

2.3. !he I!" "rchitecture

2.3.1. !he I!" (oncept 0odel

The -T' is base# on the i#ea that st!#ents are learnin% concepts an# sills that allo& them to

solve problems. The teachin% of a specific sill or set of sills involves m!ltiple t$pes of content

that is %ro!pe# into a earnin% nit )*. The earnin% nit #escribes the sill)s*, -nstr!ctional

'ssets )-'*, an# roblem Gro!ps )G*.

(ills are learne# via -nstr!ctional 'ssets )-'*. Three primar$ t$pes bein% !se# are te7t, vi#eo,an# practice problems. Te7t assets are materials that !se &or#s an# ima%es to provi#e

instr!ction. Vi#eo assets are m!ltime#ia presentations that the st!#ent can &atch an# listen to

%ain an !n#erstan#in% of the sill to be learne#. ractice problems are problems that allo& ast!#ent to practice the sill bein% learne#. ractice problems t$picall$ provi#e fee#bac

re%ar#in% ho& the st!#ent is or isn8t learnin% the sill. '##itional -' t$pes ma$ be a##e#.

:aster$ of a sill is #emonstrate# via problem solvin%. ' roblem Gro!p is a %ro!p of problemsthat test one or more sills. 5ach problem attempt creates a res!lt that can be lo%%e# for later

anal$sis.

2.3.2. !he I!" "rchitecture

The -T' is the central controller for learnin% activit$. The -T' re%isters #ata re%ar#in% earnin%

nits an# earnin% Conte7ts. ' earnin% Conte7t is re!este# from the -T' prior to the start of


23/88

a earnin% 'ctivit$. ' earnin% 'ctivit$ is an event or %ro!p of events lie &atchin% a movie,

rea#in% a lesson, &orin% a problem %ro!p, or an$ combination of the previo!s events combine#.

The earnin% Conte7t is active for the #!ration of the earnin% 'ctivit$. "!rin% the earnin%'ctivit$ the st!#ents &ill participate in an event or events that are tie# to a partic!lar

-nstr!ctional 'sset. These events &ill be lo%%e# &ith the earnin% Conte7t an# &ill contain the

-nstr!ctional 'sset i#entifier, &hat occ!rre# )i.e. &atche# a move#*, the res!lt if there &as one)i.e. correct*, an# the #!ration if #esire#.

Hnce the earnin% 'ctivit$ is over the earnin% Conte7t can be stoppe#. The -T' &ill tae the

res!lts lo%%e# b$ the earnin% Conte7t an# pass that #ata to the 'nal$sis 5n%ine. The en%ine can

process the #ata an# incorporate it into its e7istin% #ata. 't some point, the -T' ma$ be ase# for

t!torin% s!%%estions. The -T' can !er$ the 'nal$sis 5n%ine for recommen#ations. Hne ormore -nstr!ctional 'ssets ma$ be ret!rne# that can be presente# to the st!#ent for revie&.

2.3.3. !he I!" Object 0odel

Fi%!re 3 is a hi%h level representation of the ob=ect mo#el. -t represents ho& the

-T'5n%ine:ana%er is at the core. -t #oesn8t sho& the other classes that are also behin# thescenes connectin% the pieces to%ether.

Figure 3 8T" igh /eel %lass Diagram

2.3.3.1 I!" "nal$sis nine

The -T' 'nal$sis 5n%ine is base# on a simple al%orithm. -t &ors b$ looin% at the n!mber of

incorrect problems that are lo%%e#. 'n$time a problem is misse#, the sill that it8s attache# to

&ill be !se# to recommen# an$ associate# -nstr!ctional 'ssets.

2.3.3.2 I!"nine0anaer

The -T'5n%ine:ana%er is the primar$ interface for the #eveloper. -t mana%es the #ata an#

provi#es hoos to create assessments an# %et assets recommen#ations for t!torin%.


24/88


25/88


26/88

Figure "ssetRecord %lass

2.3.3.6 S>illRecord

The (illRecor# class hol#s i#entification information abo!t a sill an# &hether or not it iscomplete.

Figure 10 #ill Related %lasses

2.3.3.- +roblemRec

The roblemRec class contains information abo!t a problem. The sill-# lins the problem to'ssets that can be !se# to t!tor a st!#ent. (tate information abo!t !sa%e provi#es information to

the 'ssessment >!il#er that is !se# to #etermine &hen a problem ma$ be sho&n. The

'ssetRecommen#ation en%ine !ses the #ata as &ell to #etermine &hat assets are sho&n.

Figure 11 roblemRec %lass

2.). I!" "pplication +rorammer,s Interface %"+IThe follo&in% fo!r sections #escribe the process of &orin% &ith the -T' 5n%ine b$ provi#in%

state #ia%rams. The states in the #ia%ram t$picall$ #escribe an '- call. The act!al '- call is

provi#e# in the #escription of each state belo& the #ia%ram. The -T' '- #oc!mentation that is%enerate# via #o7$%en &ill incl!#e all of the parameter information nee#e# b$ the #eveloper to

!se the librar$ f!nctions.


27/88

2.).1. I!" InitialiCation

The -T' -nitiali9ation process #escribes ho& the -T' 5n%ine is instantiate# an# the #ata isloa#e#. The process of loa#in% the #ata from files or other means is left to the !ser of the

-T'5n%ine.

Figure 12 8T" 8nitiali6ation

The 4Create -nstance of -T'6 is the &here the #eveloper instantiates an instance of the-T'5n%ine:ana%er class. The specific call !se# is the new DefaultITAEngineManagerThis

class implements the interface specifications of the 4ITAEngineManagerclass. No parameters

are passe#.

The ne7t fo!r states #escribe the process of loa#in% #ata into the -T'5n%ine:ana%er . The$

co!l# occ!r in an$ or#er.

'## (ills loa# sill information into the -T'5n%ine:ana%er. The specific metho# is

ITAEngineManager.AddSkill.

'## T!tor 'ssets 'ssociate# &ith (ills is the interface that allo&s learnin%/t!torin% assets to beassociate# &ith a sill via this call. This maes it possible for the t!torin% en%ine to present the

list of items the st!#ent nee#s to !se to learn &h$ the$ faile#. The specific metho# call is

ITAEngineManager.AssociateAssetWithSkill.

'## 'ssessment roblems loa#s the problems that are !se# to assess a st!#ent8s no&le#%eabo!t a specific sill. hen the problems that are !se# assess the st!#ent no&le#%e are loa#e#

historical information abo!t performance can be incl!#e# as &ell. This helps to fine to the

recommen#ations provi#e# later. The specific call that is !se# is

ITAEngineManager.AssociateAssessmentProlemWithSkill.

in (pecific T!tor 'ssests &ith roblemsE -tems that can be !se# to reme#iate no&le#%e for aspecific problem are connecte# to a problem via this call.


28/88

2.).2. "ssessment (reation

art of the responsibilities of the -T'5n%ine is to mana%e the problems provi#e# to the st!#ent.'n assessment is a %ro!p of problems an# a challen%e that is presente# to the st!#ent. To

%enerate an 'ssessment a factor$ pattern is !se#. The Assessment!uilderis the factor$.

hat tri%%ers an assessment creation is in#epen#ent of the %ame. For the :ath Game,

assessment creation occ!rs &hen the st!#ent &als !p to a character that has an Hrb in#icatin%

that a ne& problem/challen%e is to occ!r an# clics the character. These are controlle# via o!rplot/stor$ control meas!res an# the flo& of o!r math no&le#%e. o& this is tri%%ere# is

#epen#ent on the specific %ame/stor$ #evices that are the %ame #esi%ner/#eveloper #eci#es !pon.

Figure 13 "ssessment %reation

The first step of the process is to Get from -T' an# 'ssessment >!il#er. TheITAEngineManager."reateAssessment!uilder metho# is !se# to an Assessment!uilder

ob=ect.

The (pecif$ roblem :ae!p states is #escribes the n!mber of problems an# %ro!p of problems

to be !se#. The specific metho# invoe# is Assessment!uilder.AddProlems.

' challen%e is the final !estion that is !se# to test a st!#ent8s no&le#%e. The (et Challen%e

process specifies the assessment challen%e. The metho# !se# is

Assessment!ilder.Set"hallenge.

The final step in the process is to >!il# an 'ssessment. This creates Assessmentob=ect that is!se# to %et problem #ata. The metho# call is Assessment!uilder.!uild.


29/88

2.).3. "ssessment

's state# previo!sl$, the 'ssessment is ho& the st!#ent8s no&le#%e is teste#. The 'ssessmentprocess is more comple7 than the others. The #eveloper &ill str!ct!re the %ame to present

problems an# receive inp!t from the st!#ent &hile interactin% &ith the -T' 5n%ine.

Figure 1+ "ssessment

Get (ill ist is an opport!nit$ to obtain a list of sills bein% !se# in the 'ssessment. -t can be

obtaine# from Assessment.#etSkill.

The step Get roblem (e!ence ret!rns the list of problems that are bein% !se# for assessment inthe or#er that the$ are to be presente#. The metho# for this is

Assessment.#etProlemSe$uence.

Get Challen%e roblem allo&s the #eveloper to %et the challen%e problem !se# for final testin%.

The call is Assessment.#et"hallengeProlem.

'fter the #eveloper has obtaine# the nee#e# information to perform the assessment it8s necessar$

to present the problems to the !ser. The state GameE resent roblem is &here the #eveloper&o!l# #o this.

The :ar roblem (tarte# allo&s the !ser to in#icate to the librar$ that the problem has beenstarte#. This is !se# for meas!rin% timin% #ata. The f!nction call is 4Assessment.MarStarted.

The #eveloper &o!l# allo& for inp!t from the st!#ent #!rin% GameE Tae -np!t.

'fter the st!#ent has provi#e# inp!t, it is !p to the %ame #esi%ner/#eveloper to #eci#e ho& to

score the inp!t. -f it is a m!ltiple+choice !estion, obvio!sl$ onl$ the proper selection)s* &o!l#

score as correct. "eci#in% if an ans&er is correct or incorrect is in#epen#ent of the -T' 5n%ine.

-t is #epen#ent !pon the problems an# the associate# inp!t capabilities.

:ar roblem Complete#, Correctness, Re!est Callbac for Reme#iation/Terminate'ssessment is &here #ata re%ar#in% problem completion is store# &ith the -T' 5n%ine.


30/88

The problem is mare# complete#, &hether or not it is correct, a callbac is re%istere# an# the

assessment is potentiall$ terminate#. The callbac stores if the st!#ent has faile#. The metho#

re%ister this information is Assessment.Mark"om%leted.

'fter completion of the assessment it sho!l# be close #!rin% the Close 'ssessment process. -t8s

a sin%le call that allo&s for statistics to be ret!rne#. The metho# for this is Assessment."lose.

2.).). Recommendation +rocess

Hnce an assessment has been complete#, there are t&o possible o!tcomes for the st!#ent. The

st!#ent has passe# or faile#. -f the st!#ent has passe#, there8s no nee# for f!rther teachin% abo!t

a specific sill. o&ever, &hen the st!#ent has faile# it8s necessar$ to provi#e fee#bac an#assets that can be !se# to teach the st!#ent. The Recommen#ation rocess is &here this occ!rs.

The -T' 5n%ine no&s &hat the st!#ent has not mastere# base# on the 'ssessment. sin% this

information, the&ecommendationEngineanal$9es the #ata an# recommen#s specific items.

Figure 1* Generating Recommendations

The #eveloper has #etecte# fail!re base# on the information provi#e from the 'ssessment

process in (t!#ent Faile# 'ssessment an# Callbac ret!rn. This is tri%%ers the process of

t!torin%.

The #eveloper &ill %o thro!%h the process of Re!est -T' 'sset Recommen#ation 5n%ine. 're!est is ma#e for access to the Asset&ecommendationEnginevia

ITAEngineManager."reateAsset&ecommendationEngine.

The #eveloper Get ist of T!tor 'ssets. Re!est a list of t!tor assets from the Recommen#ation

5n%ine in or#er of relevance via Asset&ecommendationEngine.#et&ecommendations'or.


31/88

The #eveloper provi#es the assets via the GameE resent 'sset ist or 57it. The %ame presents a

list of assets. )"epen#in% on %ame set!p the !ser ma$ have the option of e7itin% first or after

revie&in% some of the material.*

The #eveloper &ill nee# to present the act!al assets #!rin% GameE resent 'sset to (t!#entE The

%ame presents assets to the st!#ent for revie&.

'fter the st!#ent has revie&e# the material the a :ar 'sset Complete#. This in#icates that theassets has been previo!sl$ vie&e# an# that it ma$ not be as relevant later. The call is

Asset&ecommendation.Mark"om%leted.

(t!#ent ma$ e7ist from t!torin% &henever the %ame allo&s.

2.*. IS# #essons #earned

The -( #evelope# !n#er this contract has res!lte# in a val!able baseline for a %ame+base# -T'implementation. The process follo&e# to e7tract the embe##e# t!torin% capabilities from the

'TT :ath Game prove# s!ccessf!l in the creation of a mo#!lar an# %enerali9e# t!torin%a%ent. -n consi#eration of the %eneralit$ an# abstractness of the -T' '-, it prove# to beimperative to !se a!tomate# an# ri%oro!s test tools an# proce#!res for !nit+level, f!nctional, an#

inte%rate# testin% of the -(. The res!ltin% !se of the -( in the conte7t of the sample :orse

Co#e trainer helpe# provi#e an initial vali#ation of the -T' #ata mo#el !se# to str!ct!reno&le#%e, sills, an# assessment/challen%e problems across #ifferent s!b=ect matter #omains.

"evelopin% the -( !sin% open stan#ar#s, open so!rce technolo%$ &as a e$ contrib!tor to the

s!ccess in #evelopin% an -( that is compatible &ith m!ltiple operatin% s$stems an# comp!terplatforms. -t is o!r belief that the -( &ill also prove to be compatible &ith m!ltiple %ame

en%ines.

Hne of the most important barriers that &e have learne# in !sin% the :ath Game an# -T'

technolo%ies to s!pport school/classroom settin%s is the limitations of school comp!ters ? most

have mar%inall$ a#e!ate or ina#e!ate har#&are an# operatin% s$stem confi%!rations to s!pportthe !se of 3" %ame+base# learnin% applications. This is especiall$ tr!e for %ame/sim!lation

applications or heav$ &eb clients re!ire# to perform e7pensive C an# %raphics processin% on

the native har#&are. The -( an# -T' implementation #oes a %oo# =ob at mana%in% memor$re!irements. 's the si9e an# sophistication of the -T' an# %ame %ro&s the limitations of

r!nnin% on school har#&are ma$ be too %reat. The -T' performance an# optimi9ations &ere

caref!ll$ controlle# #!rin% -( #evelopment an# this &ill contin!e to be the case as the 'TT

team evolves -T' capabilities in spite of :oore8s a&.

e have learne# the importance of havin% an -T' implementation s!fficientl$ parameteri9e# to

control the pacin% an# pro%ression of st!#ents thro!%h the assessment an# learnin% c!rric!l!m.e have learne# that the -T' also nee#s to be t!nable to control the s!ccess/fail!re threshol#s

that ens!re pro%ression base# not =!st on st!#ent nee#s b!t also their intent for !sin% the %ame)e.%., learnin%, reme#iation, or testin%*. Thro!%h ri%oro!s !se of the :ath Game an# -T' as an

a!%mentation to s!pport classroom+instr!ction, &e have been able to vali#ate the -T' #ata

collection an# meas!rement techni!es &e implemente# an# the !tilit$ of the %ran!lar #ata &e

save an# mae available for reportin% st!#ent interactions, performance, an# !se of thetechnolo%$.


32/88

H!r e7perience %aine# &hile &orin% on this pro=ect is also a res!lt of observin% specific

reactions b$ st!#ents !sin% the :ath Game &ith the embe##e# -T'. For e7ample, it is not a

%oo# i#ea to #evelop an# confi%!re an -T' that provi#es too man$ opport!nities for a st!#ent to4opt o!t6 of an intervention, the !se of learnin% content, or even from completin% a t!torial. This

is more salient in the cases &here a st!#ent is more intereste# in pla$in% the %ame )or %amin% the

s$stem* than the$ are in learnin% thro!%h !se of the %ame.

The c!rrent -T' al%orithms are not sophisticate# in terms of their intelli%ence or abilit$ to a#aptto st!#ent nee#s. o&ever, the al%orithms an# approaches !se# in the -T' have proven to be

capable of provi#in% basic st!#ent monitorin%, fee#bac, an# intervention. "!rin% -T'

#evelopment an# eval!ation, &e have learne# the importance of imme#iate st!#ent fee#bac

!sin% clear an# meanin%f!l interface controls to re#!ce st!#ent fr!strations #!rin% their learnin%an# assessment process. e have observe# that the interest an# en%a%ement levels of $o!n%

st!#ents to learn !sin% o!r math %ame is #!e as m!ch b$ their e7citement, interest, an#

familiarit$ in pla$in% %ames, as it is from the !se of an$ clever or sophisticate# t!torin% en%inethis incl!#es the interest an# motivation create# b$ %oo# stor$, #ialo%, character fi#elit$, an# the

relevance of the %ame &orl# to st!#ents8 realit$ )i#s lie sateboar#s ? i#s &ant to see

sateboar#s*.

2.. 'uture Directions

H!r team has i#entifie# an arra$ of f!t!re R" p!rs!its to enhance the -( an# mae it more

!sef!l an# a#aptive for a broa#er set of s!b=ect matter #omains. Hne of the first enhancements is

to e7ten# the c!rrent set of -T' timers that have been implemente# to s!pport en%a%ementmo#elin%. The ob=ective is to #evelop a set of he!ristics that #etect #eficiencies in st!#ent

attention, bore#om, or an$ other in# of st!#ent #isen%a%ement from learnin%. sin% the proper

inferences, the -T' co!l# be mo#ifie# to tri%%er a s$stem of re&ar#s )e.%., %ame pla$, mo#ifie#learnin% tra=ectories* an#/or alerts help the st!#ent re+en%a%e &ith the %ame an# t!tor.

'nother opport!nit$ for research is the #evelopment of an -T' mo#el to create intrinsicrepresentations of c!rric!l!m paths an# trac st!#ent preferences as it relates to specific learnin%

content or %ame pla$ interests. -t mi%ht be possible to act!all$ !se #irecte# %raphs to

control/constrain st!#ent pro%ression thro!%h the c!rric!l!m an# %ame pla$. 5ach no#e in the%raph co!l# represent specific topics &ithin a concept or !nit. -t also ma$ be possible to i#entif$

cl!sters of no#es that correspon# to c!rric!l!m concept )!nits* that are most beneficial to the

st!#ents learnin% e7perience then constrain the st!#ent to follo& paths thro!%h those cl!stersan#/or no#es.

-t is important to eep %ame frame rates !p to ens!re a !alit$ learnin% an# %amin% e7perience

for the st!#ent. o&ever, as the -T' an# e#!cational %ame applications become moresophisticate# or comple7, it ma$ become necessar$ to consi#er r!nnin% -T' f!nctions o!tsi#e the

%ame8s processin%/memor$ space. This nee# creates an R" opport!nit$ to #evelop o!t+of+%ame anal$sis mo#els for #$namic t!torin% control to incl!#e o!t+of+%ame t!torin% a%ents

operatin% on local caches of st!#ent performance #ata an#/or clo!#+hoste# -T' server processes

to operate on remote caches an# persistent #atabases of st!#ent performance.


33/88

'nother potential area of research is effective pre+test capabilities an# metho#s that &o!l# assess

st!#ents e7istin% no&le#%e or sills an# allo& st!#ents to potentiall$ 4test o!t6 of certain topic

areas, set #ifferin% levels of #iffic!lt$ of challen%e an#/or practice problems, or some other#ifferentiate# learnin% capabilit$ that co!l# be !se# to control both the se!encin% an# content

#eliver$ of learnin% materials. ' relate# research area &o!l# be to #evelop a scaffol#e# learnin%

capabilit$ to provi#e ob=ective+base# content #eliver$ an# challen%e problems #riven b$ >loom8scate%ories.

Hther research ob=ectives in #evelopment of a an -( >!% "atabase creation capabilit$ base#

s!b=ect matter #omains the %oal is to create techni!es for efficient #ata minin% an# f!sion of

st!#ent #ata bases to #evelop mo#els for representin% the most common st!#ent problems an#

fail!res. The ob=ective &o!l# be to allo& the -T' to infer ca!ses of st!#ent fail!re an#/orproactivel$ %!i#e st!#ents before fail!re.

astl$, -T' capabilities for a!tomatic test %eneration are nee#e# a more #iffic!lt challen%e is to

%enerate a #omain in#epen#ent frame&or for a!tomatic %eneration of assessment an# challen%e

problems. The nee# is to move the c!rrent -T' characteristics of loa#in% static problems to acapabilit$ to provi#e a self+%enerate# assessment an# challen%e problems. ' %oal &o!l# be to

create an -T' that &o!l# provi#e assessments base# on st!#ent competencies ? proficient

st!#ents are provi#e# more challen%in% problems. ' relate# an# har#er capabilit$ &o!l# be to

%enerate an# anal$9e 4incorrect problems6 an# trac st!#ents8 abilit$ to resolve incorrectproblems.

'n$ a#vances in the research areas =!st #escribe# &o!l# provi#e si%nificant improvements to the

-( b$ main% the -T' more a#aptive an# effective for in#epen#ent or assiste# learnin%. The -(

baseline #evelope# !n#er this contract provi#es an# e7tensible soft&are frame&or to s!pportthis research.


34/88

3 8T"@!ath Game ;aluations

3.1. Introduction

-n an a%e of technolo%$ an# commerciall$ available vi#eo sim!lation %ames, to#a$8s st!#ents are

!se# to %amin% an# learnin% environments that are fast+movin%, self+#eterminin%, #eman#in%,

%raphicall$ oriente#, an# technolo%$+#riven. These e7periences lea# to st!#ents8 e7pectations oflearnin% as bein% f!n, motivatin%, an# active. 't the same time the st!#ent pop!lation is %ettin%

har#er to teach an# motivate &ith tra#itional approaches, o!r teacher pop!lation is a%in%. These

teachers are steepe# in val!able content no&le#%e b!t nee# ne& pe#a%o%ical strate%ies to meetthe nee#s of to#a$Js learners. '#vance# Trainin% an# earnin% Technolo%ies )'TT* create#

the %ame, i an# The ost F!nction, as a tool for st!#ents to learn re+'l%ebra content in a

virt!al &orl# &ith s!pport of an -ntelli%ent T!torin% '%ent )-T'*. This report is a #escription of

the formative eval!ation of the %ame &ith teachers an# st!#ents from a lar%e cit$ p!blic school#ivision.

3.2. +urpose

>e%innin% in Febr!ar$, 2012, 'TT recr!ite# local mi##le school mathematics teachers )fromseveral #ifferent p!blic school #ivisions an# in#epen#ent schools* to provi#e fee#bac in or#er to

improve the -T', !se of the %ame, an# to eval!ate &hether the %ame &as meetin% %oals of

instr!ction an# potential learnin% o!tcomes. 'fter !sin% #ata from the initial eval!ation to maeimprovements, in 'pril an# :a$, 2012, 'TT calle# bac some of the same participants from

the first phase of the formative eval!ation, an# a##e# 1; teachers from a mi##le school that the

local p!blic school #ivision i#entifie# as a f!t!re ven!e for st!#ent eval!ation. This partic!lar

mi##le school has !se# (T5: pro%rams in the past for st!#ent learnin% an# aca#emicinterventions. "!rin% the secon# phase of formative eval!ation, teachers contin!e# to provi#e

fee#bac relate# to !sabilit$ of the %ame, an# a##itional emphasis &as foc!se# on teachers8

fee#bac relatin% to instr!ctional processes an# learnin% o!tcomes. 'fter !sin% #ata to maea##itional improvements, it &as appropriate to %arner st!#ent fee#bac. "!rin% Q!l$ an# '!%!st,

2012, 'TT co+hoste# 2 mi##le school math camps in or#er to eval!ate teachers8 perceptions of

!sabilit$ of the %ame as an instr!ctional tool, an# st!#ents8 liabilit$ an# mathematicsperformance in a formal st!#$. Formative eval!ation has been on%oin% thro!%ho!t the life c$cle

of the soft&are &ith the intent to improve component technolo%ies, !alit$ of !ser interaction,

an# the potential of the pro%ram to achieve learnin%.

3.3. 0ethodolo$ and Research 4uestions

The formative eval!ations foc!se# on three areasE improvement of component technolo%ies,

!alit$ of !ser interaction, an# potential of the pro%ram to achieve learnin%. 5ach phase of the

eval!ation pro%ressivel$ shifte# from anal$sis of the soft&are to potential for learnin% to st!#ento!tcomes. "!rin% hase Hne of the formative eval!ation, via s!rve$, participants provi#e# #ata

relate# to controllin% the pla$er &ithin the %ame, interactin% &ith characters, hea#s !p #ispla$

)"*, an# the learnin% assets &ithin the tablet )vi#eos, te7t, an# practice problems*. >ase# onfee#bac from hase Hne, mo#ifications &ere ma#e to the %ame. "!rin% hase T&o, via s!rve$,

participants provi#e# #ata relate# to !alit$ of !ser interaction an# perceptions of the t!torin% an#

learnin% components of the %ame. The hase T&o eval!ation shifte# in foc!s to&ar# the -T' an#

the #e%ree to &hich participants vie&e# it as fosterin% learnin% an# the abilit$ to increase st!#ent


35/88

performance. hase Three of the eval!ation also incl!#e# teacher fee#bac, b!t the foc!s shifte#

to st!#ent learnin% o!tcomes an# en%a%ement. The research !estions for hase Three &ere

centere# aro!n# the #e%ree to &hich the %ame an# -T' improves re+'l%ebra performance an#teachers8 an# st!#ents8 perceptions of the %ame in terms of lieabilit$, accessibilit$, an# #esire to

!se as a t!torin% a%ent.

3.3.1. +hase One 0ethodolo$

From Febr!ar$ to :arch, 2012, ei%hteen participants &ere recr!ite# )thro!%h conveniencesamplin%* to pla$ the %ame for one ho!r an# provi#e fee#bac in the form of a s!rve$ an#

ro!n#table #isc!ssion )(ee 'ppen#i7 1*. articipants incl!#e# professional e#!cators from the

Ti#e&ater Re%ion of (o!theastern Vir%inia )both from p!blic an# in#epen#ent schools N X B*an# from Ta9e&ell Co!nt$ !blic (chools )NX ;*. articipants receive# monetar$ compensation

in e7chan%e for eval!ation participation. -n a##ition, e#!cational a#ministrators an# researchers

from the "epartment of "efense 5#!cation 'ctivit$ )"o"5'* an# the Vir%inia "epartment of

5#!cation )V"H5* participate# in the initial eval!ation )NXase# onfee#bac from hase Hne, chan%es &ere ma#e to the soft&are &ith emphasis on !alit$ an# ease

of !ser interaction an# some a#=!stments &ere ma#e to the -T' to enhance the opport!nit$ for

increasin% st!#ent learnin%. Thirt$+three participants from a local cit$ school #ivision eval!ate#the %ame. The participants &ere recr!ite# thro!%h convenience samplin% an# &ere i#entifie# b$

t&o hi%h school principals &ho e7presse# interest in hostin% hase 3 of the formative eval!ation.

'll participants &ere mi##le school professional e#!cators, a ma=orit$ teach math. articipantsreceive# monetar$ compensation in e7chan%e for eval!ation participation. er re!irements of

the local school #ivision, 'TT s!bmitte# a formal re!est for teacher participation that

follo&e# a revie& process b$ a Research Revie& Committee.


36/88

roce#!re. 'TT soft&are en%ineers an# instr!ctional team intro#!ce# the %ame, provi#e#

instr!ctions for interaction, an# ha# participants fill o!t release forms. articipants &ere allotte#

;0 min!tes to %o thro!%h the %ame t!torial an# pla$ thro!%h challen%es. articipants &ere ase#to interact &ith the %ame8s virt!al learnin% tablet )-T'*, &hich incl!#e# &atchin% vi#eos, rea#in%

te7t, an# practice problems. 'fter interactin% &ith the %ame, participants ha# 30 min!tes to fill

o!t a s!rve$ )'ppen#i7 2* that incl!#e# both close# an# open+en#e# items assessin% !alit$ of!ser interaction an# perceptions of the t!torin% an# learnin% components of the %ame. 'fter

completin% the s!rve$, a ro!n#table #isc!ssion &as facilitate# b$ 'TT eval!ators &here

fee#bac &as recor#e#.

:aterials. 57tra paper an# pencils an# calc!lators &ere provi#e# to participants to &or thro!%h

math problems an# tae notes #!rin% eval!ation. Tester comp!ters e!ippe# &ith hea#phones,mice, po&er strips, an# &ireless access &ere !se# to operate the %ame. (!rve$s &ere printe# on

paper an# participants !se# pencils to recor# responses.

3.3.3. +hase !hree 0ethodolo$

From Q!l$ to '!%!st, 2012, 'TT con#!cte# hase 3 of the formative eval!ation. -ncollaboration &ith the local school #ivision, 'TT create# an# con#!cte# t&o mathematics

camps at t&o mi##le schools in the cit$. 't :ath Camp 1, thirt$+fo!r mi##le school st!#ents

&ere recr!ite# )thro!%h convenience samplin%* an# at :ath Camp 2, sevent$+nine mi##leschool st!#ents &ere recr!ite# )thro!%h convenience samplin%* to participate. 5ach camp ha# :ath Camp 2 ha# Gro!p C Gro!p "*. Hn

the first #a$ of camp, st!#ents too a c!m!lative pre+test on math content that covere# D

s!btopics &ithin the cate%or$ of hole N!mbers )'ppen#i7 3*. 'fter the c!m!lative pretest,

st!#ents &ere instr!cte# )base# on %ro!p assi%nment* a short mini lesson on a partic!lar s!btopicin hole N!mbers an# then a#ministere# a mini pretest on the specific s!btopic. 'fter the mini

pretest, st!#ents ha# at least 30 min!tes to interact &ith the %ame in that partic!lar s!btopic area

)'ppen#i7 B*. >oth %ro!ps receive# instr!ction an# %ame pla$ on 3 of the s!btopics, an# thenone %ro!p )4%ro!p '6 or 4%ro!p C6* ha# a##itional instr!ction an# %ame pla$ on 2 a##itional

s!btopics )that %ro!p > an# %ro!p " #i# not receive instr!ction or %ame pla$ on*, &hile the other

%ro!ps )4%ro!p >6 an# 4%ro!p "6* receive# instr!ction an# %ame pla$ on B a##itional !nits )that%ro!ps ' an# C #i# not receive instr!ction an# %ame pla$ on*. 'fter st!#ents complete#

assi%ne# content, a c!m!lative post+test &as a#ministere# )'ppen#i7 @*.-n a##ition, teachers

&ere a#ministere# a teacher s!rve$ that assesse# !se of the %ame as an instr!ctional ai# an#

potential for st!#ent learnin% )'ppen#i7 ;* an# st!#ents &ere a#ministere# a st!#ent s!rve$ thatassesse# lieabilit$ an# lielihoo# of !sin% the %ame for learnin% )'ppen#i7 *.

:aterials. 57tra paper an# pencils, calc!lators, an# &ipe boar#s &ere provi#e# to st!#ents to


37/88

&or thro!%h math problems. ime (!rve$ )a &eb base# s!rve$ pro%ram* &as !se# to recor#

teacher an# st!#ent s!rve$ responses. Tester comp!ters e!ippe# &ith hea#phones, mice, po&er

strips, an# &ireless access &ere !se# to operate the %ame.

3.). Results

'TT incorporate# a variet$ of meas!res in or#er to obtain res!lts thro!%ho!t the life c$cle ofthe eval!ation. (ome #ata &as anal$9e# !antitativel$ an# some #ata &as anal$9e# thro!%h!alitative anal$sis. (ince most of the #ata collecte# from hases Hne an# T&o of the eval!ation

&ere in the form of s!rve$ fee#bac, #escriptive statistics &ere r!n on close# en#e# items.

here consistent tren#s in close# en#e# items &ere reveale#, the #ata #rove chan%es an#

improvements in the soft&are an# instr!ctional #esi%n of the %ame. Hpen en#e# items an# #ata%enerate# from ro!n# table #isc!ssions &as co#e# !sin% a !alitative approach, an# tren#s in

fee#bac lea# to improvin% the %ame interface an# instr!ctional #esi%n. For hase Three,

#escriptive statistics an# !antitative anal$sis &as !se# to eval!ate st!#ent performanceo!tcomes an# as appropriate to s!mmari9e s!rve$ fee#bac.

3.).1. +hase One Results

"!rin% hase Hne, ei%hteen e#!cational professionals participate# in the eval!ation. (i7

professionals from TC( )B &ho c!rrentl$ teach math*, B teachers from the Ti#e&ater Re%ion )2&ho c!rrentl$ teach math*, 2 researchers from "o"5', B a#ministrators from V"H5 )2 &ho

c!rrentl$ teach math*, an# B %overnment staehol#ers participate# in the st!#$ )NX 1


38/88

the pla$er, interactin% &ith the characters, an# the ". Respon#ents reporte# that a##in% te7t

to the c!t scenes in the vi#eos &o!l# ai# pla$ers in rea#in% #irections an# that the vi#eos sho!l#

%!i#e pla$ers to atten# to the ne7t tas. The$ consistentl$ reporte# challen%es controllin% thepla$er incl!#in% lac of fl!i# movement, overt!rns, b!ttons movin% too !icl$, an# #iffic!lt$

main% sli%ht character t!rns. -n a##ition, participants ha# problems !sin% the map, incl!#in% the

icon &as not al&a$s visible, the map is too small, an# the$ &o!l# lie the capabilit$ to 9oom.Fin#in% characters &as also challen%in% an# fr!stratin% to initiate interaction. The " co!l# be

lar%er an# &o!l# also benefit from 9oomin% capabilit$. Table 2 ill!strates the s!rve$ fee#bac as

relate# to the t!torin% an# learnin% components.

Table 3 #ure Results Related to Tutoring and /earning %omponents

(!rve$ -tems ercenta%e of Respon#ents 'ns&erin%

Wes8

esson te7t is !n#erstan#able D3O

'ble to learn from the lesson &itho!t priorno&le#%e of material


39/88

100O a%ree the %ame &ill lea# st!#ents to ac!ire fact!al no&le#%e


40/88

L!alitative fee#bac relate# to the soft&are &as co#e# b$ responses relate# to controllin% the

pla$er an# interactin% &ith the characters. Respon#ents reporte# havin% challen%es !sin% the

4'6, 4,6 an# 4"6 e$s to move the character. The$ also tho!%ht the spee# of movement &astoo slo&. -n re%ar# to character interaction, teachers reporte# that it too too m!ch time to %et

from one character to the ne7t, an# that the $ello& #ot &as conf!sin%. The$ &ere not s!re ho& to

initiate character interaction. Table B ill!strates the s!rve$ fee#bac as relate# to the t!torin% an#learnin% components for hase T&o.

Table * #ure Results Related to Tutoring and /earning %omponents: hase 2

(!rve$ -tems ercenta%e of Respon#ents 'ns&erin%

Wes8

esson te7t is !n#erstan#able D3O

'ble to learn from the lesson &itho!t prior

no&le#%e of material

O

Vi#eos provi#e eno!%h information for

st!#ent to learn &itho!t rea#in% te7t

D3O

5no!%h information in vi#eo for st!#ents to

learn

D0O

(t!#ents &ill be able to learn from the

practice problems

D3O

'ssessment !estions are eas$ to !n#erstan# D0O

'ssessment !estions are appropriate D0O

Challen%e !estions represent real &orl#

problems

D3O

For items co#e# on a B+point iert scale )(tron%l$ "isa%ree, "isa%ree, '%ree, (tron%l$ '%ree*the follo&in% ans&ere# '%ree or (tron%l$ '%reeE

DO a%ree the %ame &ill lea# to retention of information learne#

DO a%ree the challen%e !estions acc!ratel$ represent st!#ent no&le#%e ac!isition D0O a%ree st!#ents &ill be en%a%e# in a!thentic learnin%/challen%in% learnin%


41/88


42/88

articipant attrition occ!rre# beca!se participants #roppe# o!t or #i# not tae pre+tests or post+

test. The in#epen#ent variable for this st!#$ &as %ro!p membership, co#e# as either %ro!p ' or

%ro!p > ):ath Camp 1* or %ro!p C or %ro!p " ):ath Camp 2*. The #epen#ent variables inthis st!#$ &ere the D s!btopics scores on the post+test an# the c!m!lative post+test score. re+test

an# post+test scores &ere converte# to percenta%e correct.

"escriptive (tatistics

:ath Camp 1E :eans &ere calc!late# for the c!m!lative pre+test )%ro!p 'E : X B.0E : X B;.;* an# c!m!lative post+test )%ro!p 'E : X @.;1, %ro!p >E : X ;0.;@*. 'n

in#epen#ent samples ttest &as comp!te# on the c!m!lative pre+test to eval!ate &hether the t&o%ro!ps #iffere# in terms of mean score. The test &as insi%nificant t)3;* X .0,%X .DB, &hich

meant there &ere no mean #ifferences in c!m!lative pre+test performance bet&een %ro!p ' an#

>, th!s the ran#om assi%nment of participants to %ro!ps &as s!ccessf!l.

:ath Camp 2E :eans &ere calc!late# for the c!m!lative pre+test )%ro!p CE : X B


43/88

-n :ath Camp 1, the s!btopics that %ro!p ' &as e7pose# to )thro!%h instr!ction an# %ame

pla$* means on post+test s!btopic scores &ere hi%her than mean s!btopic scores for %ro!p >. For

the s!btopics that %ro!p > &as e7pose# to means on post+test s!btopic scores &ere hi%her thanmean s!btopic scores for %ro!p ', &ith onl$ a sli%ht #ifference in the s!btopic of Greatest

Common Factor. hen e7aminin% the content both %ro!ps &ere e7pose# to, for ritin%

57pressions, the %ro!ps performe# similarl$. For 5val!atin% 57pressions, %ro!p > score#notabl$ hi%her than %ro!p ' an# for (implif$in% 57pressions, %ro!p ' score# notabl$ hi%her on

the post+test.

-n :ath Camp 2, at face val!e, the s!btopics that %ro!p C &as e7pose# to )thro!%h instr!ction

an# %ame pla$* means on post+test s!btopic scores &ere hi%her than mean s!btopic scores for

%ro!p ". For the s!btopics that %ro!p " &as e7pose# to means on post+test s!btopic scores &erehi%her than mean s!btopic scores for %ro!p C. hen e7aminin% the content both %ro!ps &ere

e7pose# to, the %ro!ps performe# similarl$ %ro!p C consistentl$ score# hi%her, on avera%e, than

%ro!p ".

Table ; contains the means an# stan#ar# #eviations on the #epen#ent variables for the t&o%ro!ps for :ath Camp 1 an#, Table contains the means an# stan#ar# #eviations on the

#epen#ent variables for the t&o %ro!ps for :ath Camp 2.

Table ( Descriptie #tatistics 9or ost-test #ubtopic #cores: !ath %amp &1

ost-test

subtopic scoree ;2.@0 2D.@0 1;

Total ;


44/88

ost-test

subtopic scoree 2.D2 30.3@ 1;

Total ;@.;D 3;.22 3B

(implif$in%

57pressions

%ro!p ' 2.22 2 ;0.B2 32.0 1;

Total ;;.; 30.0 3B

Re# #enotes %ro!p ' onl$ e7pose# to the s!btopic

Green #enotes %ro!p > onl$ e7pose# to the s!btopic

>l!e #enotes both %ro!ps e7pose# to the s!btopic


45/88

Table ) Descriptie #tatistics 9or ost-test #ubtopic #cores

ost-test

subtopic score

e


46/88

ost-test

subtopic scoreel!e #enotes both %ro!ps e7pose# to the s!btopic

Table D s!mmari9es the fee#bac from teacher s!rve$s from both math camps. Table D

s!mmari9es the fee#bac from st!#ent s!rve$s from both math camps.

Table Teacher #ure Feedbac

(elect (!rve$ -tems :ath Camp 1 :ean :ath Camp 2 :ean

sef!lness of the %ame in

teachin% pre+al%ebra

B.@0 B.3loom8s

Ta7onom$ mappin% as #isc!sse# in (ection 2.2.2.2. '##itionall$, 'TT collaboration &ithpro=ect staehol#ers incl!#e# provi#in% "o"5' personnel &ith -R> (!pport b$ passin% copiesof Research Re!est 'pplications !se# to ac!ire permissions to con#!ct the formative

eval!ations in a local p!blic school s$stem. ' >eta version of the :ath Game &as ma#e

available to "o"5' alon% &ith a license capable of s!pportin% the n!mber of st!#ents enrolle#

in a s!mmer enrichment pro%ram at Ramstein 'F>. The %ame &as s!ccessf!ll$ #eplo$e# &ithno problems an# !se# in an informal settin% &ith ;th+


49/88

access to pre+ an# post+test #ata/assessments. Hn%oin% meetin%s &ith school a#ministrators have

been occ!rrin% &ith the %oal of movin% to&ar# %ather school+%enerate# #ata. -n a##ition, the

%ame is in the process of bein% ac!ire# at t&o hi%h schools in the same #ivision. The hi%hschools &ant to !se the %ame as a reme#iation plan for st!#ents &ho faile# their re+'l%ebra

state test the prior $ear. The hi%h schools &ant to !se the %ame to catch st!#ents !p on pre+

re!isite no&le#%e nee#e# for their c!rrent co!rse&or in 'l%ebra -. '%ain, 'TT is inre%!lar comm!nication &ith these schools, settin% !p plannin% for #ata collection. 's of no&, it

appears that the %ame &ill be more heavil$ !se# to&ar# the en# of the sprin% semester as all

schools prepare for ann!al state testin%. 'TT hopes to have the approval in earl$ 2013 tocollect pre an# post+test #ata in the form of teacher tests, !arterl$ assessments, an# en# of $ear

state test scores.

3.6. "dditional #essons #earned

Hver the co!rse of the eval!ation, &e have contin!e# to mae improvements to the soft&are an#instr!ctional #esi%n in or#er to !se the %ame as an effective teachin% an# t!torin% tool. e have

seen teachers an# st!#ents en%a%e# an# e7cite# abo!t the %ame an# the$ have provi#e# !s &ith

i#eas to problem solve alon% the &a$. o&ever &e have fo!n# that there are several componentsthat nee# to be in place before incorporatin% the %ame into their c!rric!lar tools. First, clear

e7pectations an# clear teacher trainin% on the technolo%$ is important. e learne# that the %ame

hol#s a lot of potential as a reme#iation tool, not a s!bstit!te for teacher instr!ction. henteachers feel comfortable &ith !sin% the %ame as a s!pport, the$ are more enth!siastic to !se it.

-n a##ition, the more comfortable the$ are &ith !sin%/pla$in% the %ame, the more liel$ the$ are

to !se it &ith the st!#ents. Teachers &ho ta!%ht #!rin% the math camps are !sin% the %ame more

fre!entl$ than teachers &ho are =!st learnin% abo!t the %ame. This le# 'TT to #evelop theteacher trainin% an# teacher &orboo. Teachers #o not nee# to be hi%hl$ e7pert in the !se of the

%ame, itself, b!t the$ m!st be &ell prepare# to teach the !n#erl$in% content that is re!ire# b$

the %ame. "!rin% teacher trainin%s, &e have notice# that non+math teachers are less en%a%e#, an#

ma$ be less liel$ to !se the %ame #!rin% reme#iation labs. -f &e can enco!ra%e non+mathteachers to e7plore the %ame an# interact &ith the tablet, the$ can also learn from the t!torin%

a%ent an# then assist st!#ents. e have also learne# that teachers are !n#er a lot of press!re toreport #ata base# on st!#ent performance. -n response, 'TT has #evelope# a Teacher ortal,

&hich is a reportin% tool. e are c!rrentl$ in the process of enhancin% this tool so that it is !sef!l

to teachers an# a#ministrators. e are con#!ctin% reportin% re!irements anal$ses an# have

plans to improve the Teacher ortal b$ earl$ 2013.


50/88

+ Re9erences

).1. "cron$ms

"8? 'pplication ro%rammin% -nterface

"T./T? '#vance# Trainin% earnin% Technolo%ies, C

%#%8? Comp!ter (oft&are Confi%!ration -tem

8#/? -T' (oft&are ibrar$

8T"? -ntelli%ent T!torin% '%ent

! TR"#$#? ro%ram :ana%er Trainin% ($stems

!/? 57ten#e# :ar!p an%!a%e

).2. References

1S '#ams, 5. )2010*. F!n#amentals of Game "esi%n, 2n#5#. Ne& Ri#ers. >erele$, C'.

2S 'l#rich, C. )200D*. The Complete G!i#e to (im!lations (erio!s Games. feiffer. (an

Francisco, C'.

3S >loom, >. )1D, TI.

.. )200D*. >!il#in% -ntelli%ent -nteractive T!tors. :or%an Aa!fmann. >!rlin%ton,

:'.


51/88

"ppendi


52/88

2. -f no above, &hat #o $o! thin &as missin% from the vi#eo or ho& mi%ht it &or better

for $o![ )allo& inp!t*

Tablet Tutorial

1. "i# $o! revie& the Tablet T!torial[ es / No

2. -f no, sip to ne7t section.

3. "i# $o! !n#erstan# ho& to !se the tablet[ es / No

B. hat parts of the tablet t!torial #i# $o! not !n#erstan#[ )allo& inp!t*

General la

%ontrolling laer

1. "i# $o! have challen%es movin% the character[ es / No

2. -f $es to the above, &hat in# of challen%es #i# $o! have[ )allo& inp!t*

3. "i# $o! no& that $o! co!l# mae the character r!n b$ pressin% the (hift Ae$ K p'rro&[ es / No

B. "i# $o! no& that $o! co!l# chan%e the oint of Vie& )HV* b$ pressin% an# hol#in%the left mo!se b!tton an# movin% the mo!se[ es / No

@. -f $es to the above, #i# $o! chan%e the HV of the character[ es / No

;. "i# $o! have an$ problems navi%atin% !sin% the HV presente#[ es / No

. -f $es, &hat challen%es #i# $o! have[ )allo& inp!t*

8nteracting ?ith %haracters

1. ere $o! able to fin# the characters that $o! &ere s!ppose# to interact &ith easil$[ es /No

2. -f no to the above, #escribe the challen%es that $o! ha#E )allo& inp!t*

3. "i# $o! attempt to clic the re# cr$stal before clicin% the character[ es / No

eads >p Displa 4>D51. as the " in the !pper left han# corner helpf!l[ es / No

2. "i# $o! reali9e that the ne7t challen%e location &as a $ello& #ot[ es / No

3. ere $o! able to !se the " to navi%ate aro!n# the %ame[ es / No


53/88

Tablet

/esson Te


54/88

%haracter 8nteraction roblems

Iarm >p Huestions

1. ere the !estions eas$ to !n#erstan#[ es / No

2. ere the$ appropriate to the material that &as covere# in the tablet[ es / No

3. as there an$thin% that co!l# be improve# abo!t the !estions[ es / No

B. -f $es, please provi#e fee#bac. )allo& inp!t*

%hallenge Huestions

1. "o the challen%e !estions feel that the challen%e !estions represente# real &orl#problems[ es / No

2. -s there an$thin% that co!l# be improve#[ es / No

3. -f $es, please #escribe. )allo& inp!t*

erall Feedbac

lease in#icate the e7tent to &hich $o! a%ree or #isa%ree &ith the follo&in%E

1. The %ame &ill lea# to retention of information learne#

'* (tron%l$ '%ree >* '%ree C* "isa%ree "* (tron%l$ "isa%ree

2. "o $o! feel that the assessment !estions in the %ame acc!ratel$ represente# st!#ent

no&le#%e ac!isition[

'* :ost of the Time >* (ome of the Time C* Rarel$ "* Never

3. The %ame implements learnin% strate%ies/theories


B. The %ame implements learnin% strate%ies/theories


@. (t!#ents &ill be en%a%e# in a!thentic/challen%in% learnin%


;. >ase# on the a%es of st!#ents $o! &or &ith, #o $o! feel the %ame is #evelopmentall$appropriate[



55/88

. ill the st!#ents !n#erstan# the lan%!a%e !se# in the %ame script[



56/88


57/88

e. mathematics grade le$el )

f. algebra I honors middle school#

g. geometry honors middle school#

h. high school le$el mathematics

i. other

*. What math subjects ha$e you taught in the "ast ans!er all that a""ly#?

a. mathematics grade le$el %

b. mathematics grade le$el &

c. "re'algebra

d. mathematics grade le$el (

e. mathematics grade le$el )

f. algebra I honors middle school#

g. geometry honors middle school#

h. high school le$el mathematics

i. other

Skipto question %

4. If you are not currently a math teacher+ !hich of the follo!ing best describes your

current role,a. -echnology -eacher

b. anguage Arts -eacher

c. /cience -eacherd. 0hysical ducation -eacher

e. /ocial /ciences -eacherf. /chool Administrator

g. ther

%. What grade le$els# do you currently teach3!or !ithin,a. lementary /chool

b. 5iddle /choolc. 6igh /chool

&. What7s the highest le$el of math course that you ha$e taen?


58/88


59/88

2. yes

2. If no+ !hat did you not understand in ho! to use the tablet? And+ do you ha$e

suggestions for ho! to im"ro$e the tutorial to aid understanding? allo! in"ut#

%ontrolling laer

1. In the beginning of the game+ you !atched a tutorial that described ho! to

na$igate3mo$e the character. =id this tutorial effecti$ely e;"lain ho! to mo$e thecharacter in the game?

1. no

2. yes

2. =id you ha$e challenges mo$ing the character?

1. No

2. yes

*. If yes to the abo$e+ !hat !ere your challenges? And+ do you ha$e suggestions for

ho! to im"ro$e the tutorial to aid understanding? allo! in"ut#

8nteracting ?ith %haracters

1. Were you able to find the characters that you !ere su""osed to interact !ith easily?

1. no

2. yes

2. If no to the abo$e+ describe the challenges that you had, allo! in"ut#

*. =id you attem"t to clic the yello! s"here before clicing the character?

1. no

2. yes

!ap

1. =id the circular ma" in the u""er left hand corner of the screen hel" you no! !hereto go ne;t?

1. no

2. yes

2. =id you reali>e that the ne;t challenge location !as a yello! dot?


60/88

1. no

2. yes

Tablet

o! &ere ase# to f!ll$ interact &ith the tablet for a challen%e &ithin the %ame. The tablet offersoptions for rea#in% lesson te7t, &atchin% a vi#eo, or &orin% on practice problems. lease

ans&er the follo&in% !estions base# on $o!r time interactin% &ith each of the components.

/esson Te


61/88

%. =o you ha$e suggestions for ho! the lesson te;t could be im"ro$ed? allo! in"ut#

ideos

1. hich vi#eo)s* #i# $o! &atch )ans&er all that appl$*[

a. sin% the #ivisibilit$ r!les

b. -#entif$in% %reatest common factor

c. istin% factors

#. -#entif$in% properties of a##ition m!ltiplication

e. 5val!atin% e7pressions an# or#ers of operations

f. ritin% e7pressions

%. 5val!atin% e7pressions

h. '##in% inte%ers &ith lie si%ns

i. '##in% inte%ers &ith !nlie si%ns=. -#entif$in% inte%ers. -#entif$in% opposite inte%ers an# absol!te val!e

l. Graphin% inte%ers

m. Comparin% inte%ers

n. :o#elin% real life !sin% inte%ers

o. :!ltipl$in% an# #ivi#in% inte%ers

p. Combinin% inte%er m!ltiplication #ivision

. (!btractin% inte%ers

r. Combinin% inte%er s!btraction a##ition

2. =id the $ideos "ro$ide enough information in combination !ith the esson -e;t# toallo! you to understand !hat you !ere ased to learn?

1. no

2. yes

*. Was there enough information in the $ideo to allo! a studentto learn the

information !ithout reading the lesson te;t?

1. no

2. yes

4. =o you feel that the $ideos# e;"lain the material in an easy to follo! manner forstudents?

1. No

2. Yes

%. =o you ha$e suggestions for ho! the $ideos could be im"ro$ed? allo! in"ut#


62/88


63/88


64/88


65/88

If you are currently a teacher+ ho! !ould you use the game to su""lement instruction?

allo! in"ut# If you !ould not use the game 9 "lease use this s"ace to e;"lain !hy#.

0lease "ro$ide as much general feedbac about the game as you can. allo! in"ut#


66/88


67/88


68/88


69/88


70/88


71/88

"ppendi< + hase Three Formatie ;aluation - !ini re-Test


72/88


73/88


74/88


75/88


76/88


77/88


78/88


79/88


80/88

"ppendi< * hase Three Formatie ;aluation - %umulatie

ost-Test


81/88


82/88


83/88


84/88


85/88

"ppendi< ' hase Three Formatie ;aluation - Teacher #ure


86/88


87/88

"ppendi< ( hase Three Formatie ;aluation - #tudent #ure


88/88

"ppendi< ) hase Three Formatie ;aluation - BloomJs

!apping

This table provi#es the co#in% of assessment items for the hase Three re+ an# ost+Tests base#on >loomJs Ta7onom$. The co#in% metho# &as #evelope# an# e7pan#e# b$ Qohn :a$nar#.

Documents

ATLT Final Technical Report 2012-12-31 v1