What to Consider When Developing and Evaluating Games for Training

U.S. Army Research, Development and Engineering Command

*Unclassified – Approved for Public Release*

What to Consider

When Developing and Evaluating Games

for Training

Teresita Marie Sotomayor, PhDScience and Technology ManagerUS Army Research Laboratory Simulation and Training Technology Center

SFC Paul Ray Smith Simulation & Training Technology Center

Unclassified – Approved for Public Release*

Outline

• This presentation will explore:

– Generational differences and the characteristics of the new generation of trainees

– Features of games that motivate and engage individuals

– An overview of the elements that need to be considered when evaluating the effectiveness of games in a training environment



Generational Differences• When developing games for training it is important to

consider nested generational groups within the learning population.

Generation X

• Prefer fast-paced

presentation of

information

• Frequent feedback

• Sense of

accomplishment

• Visual – grown up in

the 3Dworld

• Single channelers

Generation Z

• Born with complete

technology

• Highly creative and

collaborative

• Multi-taskers

• React fast

Generation Y

• grown up with the

internet at their

fingertips

• World to them is virtual

• Possibilities are

endless

• Multi-channelers

Training Environment



Why?

• When designing tools for training it is important to understand targeted audience:

– Characteristics

– Learning Preferences

– Cognitive Styles



Sample Population Profile

Educational background Active Duty Combat Medics and Reservists: some college

Prior level of medical training Active Duty and Reservist: some prior and combat medical training

Computer literacy level Active Duty: intermediate level computer literacy

Reservists: basic computer literacy

Male/female ratio 5 to 1 male to female

Age range 18 to 50

level 12th grade

Learning style Visual and Kinesthetic

Background Various backgrounds

Some foreign studentsMotivation for completing the training

Job requirement, advancement, job knowledge, personal pride




AgeRange: 18 to 42Generation X: 13%Generation Y: 87%

GenderMale: 73%Female: 27%Male to Female Ratio: 3 to1

ExperienceCombat Operations: 8%Combat Medic: 2%Civilian Setting: 29%




Video Game Experience: 93 %Ability: Beginner: 29% Intermediate: 48% Expert: 23%

Computer Ability: Beginner: 11% Intermediate: 72% Expert: 16%

EducationHS: 30 %Some College: 53%Associate: 10%Bachelor: 6%Post Bachelor: 1%



Age Distribution

0

10

20

30

40

50

60

70

80

18-21 22-25 26-29 30-33 34-37 38-42

Age Subsets

Num

ber

of M

edic

sNumber of Medics

Gender Generation X

Total # (Percentage)

Generation Y

Total # (Percentage)

Male 17 (12.9) 115 (87.1)

Female 6 (12.5) 42 (87.5)

Total 23 (12.7) 157 (87.3)

93%


Gender and Generation Categorization



Video Game Experience

ExperienceNo Experience

93%


Id # Age Generation Gender Education

10 26 Y Female HS

18 30 Y Female Bachelors

1 41 X Male Bachelors

26 22 Y Male Some College

8 40 X Male Associate

19 18 Y Female HS

47 19 Y Male HS

115 20 Y Female Some College

148 19 Y Female HS

138 27 Y Female Bachelors

75 23 Y Female Some College

68 26 Y Female Associate

126 35 X Male Some College

Descriptive Characteristics of Subjects

without Video Game Experience



First Person Video Game Ability

29%

48%

23%

BeginnerIntermediateExpert


GenderFirst Person Video

Game Ability

Beginner Intermediate Expert

Computer Ability

Beginner Intermediate Expert

Female 32(67%) 15(31%) 1(2%) 6(13%) 38(81%) 3(6%)

Male 21(16%) 71(54%) 40(30%) 14(11%) 91(69%) 26(20%)

First Person Video Game and Computer Ability Self Assessment



Distribution of Hours Spent Gaming

0

20

40

60

80

100

120

0-5 `6-11 `12-17 18-23 24-29 30-35 36-41 42-47 48-50

Number of Hours

Num

ber o

f Med

ics

Number of Medics


Distribution of Video Game Hours on a Typical Week



• Games are a form of fun

• Games are a form of play

• Games have rules

• Games have goals

• Games have outcome and feedback

Why Games Engage?



• Games are adaptive

• Games have win states

• Games have conflict/competition/challenge/opposition

• Games have problem solving

• Games are interactive

• Games are immersive

Why Games Engage?



• Games are Interactive

• Interactivity in Games reinforces:

- Individual Control

- Trial-and-Error

- Constant Change

Why Games Engage?



Instructional Games• Instructional Games:

– Designed or modified to meet training objectives

– Includes rules, constraints, and activities that closely replicate the constraints of the real world

tasks being trained

– Must be incorporated in the POI in a manner that ensures that trainees understand the training

objectives and receive feedback on performance against the training objectives



Instructional Games• Instructional Games:

– Should be considered as a training aid tool

– If developed as a stand-alone training tool, the game must be designed such that all instructional

capabilities provided by the trainer are incorporated

– Feedback should be incorporated as a structured guided activity such that learning can be

fostered and the experience can be meaningful to training



Instructional Games

• Four main aspects of a “serious game”:

1. Context of the game and usage is essential to the effectiveness of the game in the training

environment

2. Learner Specification – user profile to understand characteristics, learning preferences and

cognitive styles of the targeted group of learners



Instructional Games

• Four main aspects of a “serious game”:

3. Representation – how levels of immersion, fidelity, and interactivity are integrated such that it is

effective and provides the required level of engagement

4. Pedagogical model – learning theory and approaches behind the application to ensure that it is

used effectively to support specified learning



When Designing a “Serious Game”

• USER Centric Approach

- Gap Analysis and Requirement Definition

• Structured SME Discussions

• Observation of training to ID objectives and gaps in training

• Training Curriculum Review

• Task Analysis

- Refinement

• Usability Studies

- Assessment

• Training Effectiveness Evaluation



The usability of an interactive system can be defined, in part, in terms of

how easily a user can access and use the intended functionality of the

system to meet task objectives.

Usability Studies



• Interactive systems should be designed such that they are intuitive,

effective, and subjectively acceptable to users.

• Performing a usability study early on can provide valuable information for

system development.

Usability Studies



Early identification of user issues translates into reduced system redesign

costs, enhanced user satisfaction and decreased user training time and the

need for technical support.

Usability Studies



• User Centric Methodology utilized to:

- Gather user (instructors and potential users) feedback regarding the current state of the

technology being developed

- Explore the feasibility of incorporating innovative training tools in the current program of

instruction

Usability Studies



• Observation of trainees and trainers interacting with the system

• Feedback on their reactions captured through surveys and participation in

structured focus group interviews.

Methodology



• Questionnaires are effective instruments to collect:

– Demographic data

– Feedback regarding their experience with the Simulation

– Reaction to training

Methodology



• Usability questionnaires are effective instruments to gather subjective data

regarding user’s reaction:

- Benefit to Training, System Usability, Presence and Realism, and Motivation to Use

- Training Objectives, Knowledge and Skills, Overall Reaction to Training

Methodology



• Focus groups are effective instruments to gather collective subjective data that

can’t be collected via survey

- Feedback about the current state of the prototype

- Scenario/Environment, Techniques, User Interface, Training Objectives and After Action Review

(AAR)

Methodology



Training Effectiveness Evaluation

• The main purpose of a training program is to impart the acquisition of knowledge, skills, and competencies.

Evaluating a particular training is a challenging task



Training Effectiveness Evaluation

• Developed by Donald Kirkpatrick in 1952 and published in 1959 in the Journal of the American Society of Training Directors

• Provides a methodology for evaluating training programs

• It is widely used because is simple and practical and can be applied to different situations (is flexible)



Level I

Reaction Criteria

Reaction: Evaluates user affective and attitudinal response to the training program.

Focuses on self-report measures.

Instruments: Survey, questionnaires, focus groups, etc.

Level II

Learning Criteria

Knowledge Acquisition: Evaluates user’s increase in knowledge and capability as a result of the

training program.

Focuses on learning outcomes.

Instruments: pre-/post Knowledge Test, Interview or on-the-job assessments, etc.

Level III

Behavior Criteria

Skill Transfer: Evaluates changes in behavior on the job as a result of the training program.

Focuses on the job performance measures.

Instruments: self-assessments, on-the-job performance ratings, etc.

Level IV

Results Criteria

Benefits: Evaluates the effect on the organization resulting from the training program.

Focuses on long-term productivity performance measures.

Instruments: Utility Analysis Instruments, Organizational performance reports, QA Reports, etc.

Kirkpatrick's Model



Level 1 Reaction

• Measure of how participants feel about the various aspects of a training program.

• Why should we measure reaction?

- People must like a training program to obtain the most benefit.

- To ensure participants are motivated and interested in learning.



Level 1 Reaction

• Guidelines:- Determine what needs to be found out about the training

- Develop Instruments: - Questionnaires, focus groups, researcher observations on

participants reaction to the simulation

- Need to ensure that reactions can be tabulated and quantified

- Obtain honest reactions by making the sheets anonymous



Level 2 Learning

• This is a measure of the knowledge acquired, skills improved, or attitude changed due to training. In our context we are aiming to improve knowledge of concepts, principles, or techniques.

• Why?- Measure of reaction of trainees is not an indication that

learning has taken place.

- Will indicate what principles, facts, and techniques were understood and absorbed by trainees.



Level 2 Learning

• Guidelines:

- Use a Control Group

- Develop instruments to evaluate knowledge before and after training:• Pre/post test, assessments, ratings of performance

- Use 100 percent scale

- Evaluate results statistically so that learning can be proven

- Use the results of the evaluation to take appropriate action



Level 3 Behavior

• This is a measure of the extent to which participants change their on-the-job behavior because of training. It is also referred as transfer of training.

• Why?

- There is a big difference between knowing principles and techniques and properly using them on the job.



Level 3 Behavior

• Guidelines:


- Allow enough time for change in behavior to take place

- Develop quantitative instruments:• Scenario based simulations, self efficacy instruments, on-the-job

observations

- Survey or interview one or more of the following: trainees, instructors, superiors, etc




• Measure of the long term results of the training program in terms of reduced costs,

higher quality, higher productivity, lower rates of employees turnover or absenteeism,

etc.

• Why?

It can be used to support acquisition of a specific training device or to include methodology in POI.

Warning: Evaluations become more difficult, complicated and expensive. How much of the

improvements shown over time are due to training and not other variables.

Level 4 Results



Level 4 Results

• Guidelines:


- Allow enough time for results to be achieved

- May be obtained from unit performance reports, quality inspections and interviews with Senior people




?? QUESTIONS ??

[email protected]

Documents

What to Consider When Developing and Evaluating Games for Training