California Educational Research Association Annual Meeting San Francisco, CA – November 19, 2009 Richard Wainess, Rebecca Buschang, & Deirdre Kerr Using

California Educational Research Association Annual Meeting

San Francisco, CA – November 19, 2009

Richard Wainess, Rebecca Buschang, & Deirdre Kerr

Using Human Help to InformAutomated Help in Games-BasedLearning Environments

2 / 23

Our Goal

Create a self-containedpre-algebra instructional game

as platform to examinewhat works in games for learning

3 / 23

Puppetman (a pre-algebra game)

4 / 23

Current Situation

CONDUCTED DESIGN STUDYTo examine [prototype] game/instructional

features and outcomes [learning/attitude]

Two Conditions: Mathy Version and Gamey Version

5 / 23

Issue

Players needed helpnot provided by game

6 / 23

1. Captured help needed

2. Coded the help (3 ways)

3. Analyzed game/test/help data

4. Report Results

5. Discuss Next steps

Process

7 / 23

Phase 1: Capture & Code HelpID Game Help Math Help Comments

- game mechanic - other (pls note)

- basic (e.g., job aide)

- instruction - solution - other (pls note)

Had her look up "unit" and green dots. Helped her with counting number of fraction (numerator). And that 1 1/2 is same as what else? (i.e., 3/2)




1. Basics of "unit" and green dots. 2. Counting fractions (numerator). 3. How many 1/6 is 1/3?




1. Recognizing 1/2 rather than thirds. 2. Counting 1/2s.

Plus game level and helper’s initials

8 / 23

Phase 1 Coding Results

Help Type Count

Game Help: Game Mechanic 12

Game Help: Game Other 8

Math Help: Math Basic 20

Math Help: Math Instruction 30

Math Help: Math Solution 8

Math Help: Math Other 1

46 participants need help (out of approx 120 participants)• 30 needed help 1 time• 11 needed help 2 times• 4 needed help 3 times• 1 needed help 4 times

9 / 23

Phase 2: Examined Help Elements

Comments Coded As


1. Asked guiding Questions2. Prompted help seeking


1. Asked guiding Questions


1. Can’t code

1. One rater examined 120+ help elements and created rubric2. 3 rounds with 2 raters (1/3 of data each round)3. Modified/clarified/reduced rubric after round 14. Inter-rater reliabilities of 71%, 64%, 63% (100%

w/discussion)

10 / 23

Phase 1 coding resultsHelp Type Count

Can’t code 31

Asked guiding questions 17

Explained game control 10

Stated math concept in terms of game feature/mechanic 6

Gave solution step 6

Prompted help seeking 5

Explained game mechanic 4

Asked for explanation 2

Explicit math instruction 0

Stated math concept as math 0

Gave solution 0

Gave solution strategy 0

11 / 23

Phase 3: Recoded Help Elements

Comments Coded As


• Help w/ unit• Help converting fractions• Help w/ green dots• Help w/ coil size (denom)• Help w/ coil num (numer)


• Help w/ unit• Help converting fractions• Help w/ green dots• Help w/ coil num (numer)


• Help w/ coil size (denom)• Help w/ coil num (numer)

1. One rater examined 120+ help elements and created codes

2. Second rater coded using same codes (matched 49%)3. Reviewed, clarified rubric, and reached consensus

12 / 23

Phase 3 coding resultsHelp Type Count

Clarify what one “unit” is 25

Determine appropriate coil size (denominator help) 19

Convert fractions 16

Determine number of coil pieces to use (numerator help)

11

Help with moving trampolines/coils 8

General help with fractions 7

Help with what green dots meant 5

Help with placing trampolines in correct grid position 5

Help on Level 1 4

Help combining fractions 4

Help with scrolling 4

Other help 4

Determine some aspect of distance 2

Help with understanding arrows 2

Help with undo/reset button 2

Help breaking coils into smaller parts 1

13 / 23

ANALYSIS & RESULTS

14 / 23

Differences in Gameplay

One-way ANOVA on Level Reached

• Only 27% of those reaching or exceeding level 17 needed help

• Only 35% of those reaching levels 12 to 16 needed help

• 78% of those who did not reach level 12 needed help

15 / 23

Differences in Learning

Regression predicting post-scores (results of ANOVA)(Note: Four models were run. Values below are from fourth model.)

•No difference in post scores between gamey/mathy, B = .023, p = .879

•Difference in post scores between help/no help, B = -.587, p = .001

Kids who got help also scored lower on post-test, when controlling for gender, gameplay, and pre-test

•Model r2 = .490, p = .000 is a significant predictor of post-test.

Adding interaction (help & condition) was a significant predictor of post-test scores (when added gender, pre-test, gameplay, condition, help, and measures of in-game learning), p = .033

16 / 23

Differences in Learning (cont’d)

• Interaction between help/no help and condition

Those who received help did worse those who didn’t receive help (p = .001)

B = -.587 for gamey B = -.564 for mathy

Mathy with help did better than everyone (p = .033) B = 1.095 better than gamey with help B=.508 better than gamey no help B = .485 better than mathy no help

17 / 23

Help by Condition

Help Type Gamey (N) Mathy (N)

Game Help: Other (3) Game Help (5)

Math Help: Instruction Math Help (15) (10)

PHASE 1


Explained Game Controls (3) Game Help (8)

Prompted for Explanation Math Help (2) (0)

Game Solution Step Math Help (4) (2)

PHASE 2

18 / 23

Help by Condition (cont’d)


Moving Trampoline or Coil Game Help (5) (3)

Level 1 (getting started) (1) Game Help (3)

How to Scroll (1) Game Help (3)

Defining a Unit Math Help (14) (10)

How to Reset or Undo (0) Game Help (2)

Other Help with Distance (0) Game Help? (2)

PHASE 3

19 / 23

Help by Condition Trends


Math Help: Instruction Math Help (15) (10)

Defining a Unit Math Help (14) (10)

Explained Game Controls (3) Game Help (8)

Largest Ns

Condition Game Help Math Help

Gamey 1 4

Mathy 6 0

Most help by condition

20 / 23

Next Steps

Created the following help topics and instruction to address all areas needing help

1.Determining the Level’s Denominator

2.Determining the Distance Between Jump Points

3.Changing the Denominator of a Coil

4.Common Denominators

5.Reducing the Denominator

6.Least Common Denominator of Two Fractions

7.Matching the Level’s Denominator

21 / 23

Next Steps (cont’d)

• Next Steps

Validate Assumptions through Think-Alouds(beginning December, 2009)

Embed more instruction into game

As direct instruction (January, 2009)

As contextualized help/feedback

Through an in-game help system

22 / 23

Review

• Collected “help given data” during design study

• Coded data three ways

• Analyzed codes, gameplay, & test scores for themes

• Used analysis results to define instructional needs

• Created and are about to test instructional parcels

• Will begin testing embedded instruction January 2009

23 / 23

BACKUP

24 / 23

In-game Help Comparison: Level 6

Start Help Make Bucket

Place Bucket Wrong Unit End Death

Mathy We need to bounce PuppetMan 3 spaces to reach safety! Click on the big coil and scroll up on the mouse wheel to break the WHOLE into more but smaller EQUAL pieces.

The WHOLE unit (the space between the red lines) is broken into TWO equal pieces but it will take THREE of THOSE EQUAL pieces to get to the goal. Recall - two HALVES are ONE WHOLE

When you are finished loading the trampoline, place it on the blocks to help save PuppetMan!

Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the goal? Remember ONE WHOLE UNIT is the distance between the red lines.

Remember, the coils must all be exactly the SAME size when ADDING them to the trampoline. If you cannot drag a piece in, break the coils into pieces of the SAME size (same denominator).

You are mastering the art of trampoline making! Just remember, addition requires that all the coils have to be the SAME size to be put into the same trampoline!

Make sure you see how many pieces the WHOLE (between the red lines) is broken into, THEN count how many of THOSE pieces you need to get PuppetMan to the goal.

Gamey

We need to bounce PuppetMan three half-units to reach safety! Click on the big coil and scroll up on the mouse wheel to break it in half, and then drag the 3 half pieces over!

Sorry, no help on this level.

When the trampoline says 3/2, place it on the blocks to help save PuppetMan!

This is the text that appears the player places some number of trampolines onto the grid

If you cant drag a piece in, click on the trampoline and scroll the wheel to change the units! You can also use the scroll wheel to break up the coils into the right units!

You are mastering the art of trampoline making! Always remember, all the coils have to be the same size to be put into the same trampoline!

DISABLED

25 / 23

In-game Help Comparison: Level 11

Start Help Make Bucket Place Bucket Wrong Unit End Death

Mathy You know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck.

The WHOLE UNIT has been broken into THREE pieces and it looks like you need FIVE of them in total. Since you only have HALVES break them into a piece that can also be made into THIRDS.

When you are finished loading the trampolines, place them on the right blocks!

Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you change HALVES into a unit you can also use to count THIRDS?

If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need. You can only ADD IDENTICAL pieces.

Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well.

Break everything into SIXTHS to make it easier! Two SIXTHS is one THIRD and Three SIXTHS is one HALF. Then you can count both coils and how far you need to jump.

Gamey You know the goal, get PuppetMan safely across the spears. But now the WHOLE unit has been broken down differently. Check it out and good luck.

The WHOLE UNIT has been broken into three pieces and it looks like you need five of them to get to the goal (two to the first block and three from there to the goal).

When you are finished loading the trampolines, place them on the right blocks!

Have you counted the number of EQUAL pieces the WHOLE was broken into? Did you count how many of THOSE pieces you need to reach the first block? How many more do you need to reach the goal?

If you can't drag a piece in, click on the trampoline or coil and scroll the wheel to change the units into the units you need.

Genius! Sometimes, you'll have some coils leftover! Don't worry at all about that, because PuppetMan is alive and well.

Break everything into thirds to make it easier! Then make sure you have enough thirds to reach the first block and enough to reach from there to the goal.

[email protected]

Documents

California Educational Research Association Annual Meeting San Francisco, CA – November 19, 2009 Richard Wainess, Rebecca Buschang, & Deirdre Kerr Using