Serious Gaming: Research

1Challenge the future

Serious Gaming Research© Dr. Igor Mayer [email protected]

Faculty Technology, Policy and Management (TPM) & Signature Games

Delft University of technology, The Netherlands

mailto:[email protected]



References (1)1. Mayer, I. S., Bekebrede, G., Harteveld, C., Warmelink, H. J. G., Zhou, Q., van Ruijven, T., … Wenzler, I. (2013). The

research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational

Technology, n/a–n/a. doi:10.1111/bjet.12067

2. Mayer, I. S., Bekebrede, G., Warmelink, H. J. G., & Zhou, Q. (2013). A Brief Methodology for Researching and Evaluating

Serious Games and Game-Based Learning. In T. M. Connolly, L. Boyle, T. Hainey, G. Baxter, & P. Moreno-Ger (Eds.),

Psychology, Pedagogy and Assessment in Serious Games (in press) (pp. 357–393). IGI Global. doi:10.4018/978-1-4666-

4773-2.ch017

3. Mayer, I. S., Kortmann, R., Wenzler, I., Wetters, Á., & Johan, S. (2014). Game-based Entrepreneurship Education:

Identifying Enterprising Personality, Motivation and Intentions amongst Engineering Students. International Journal of

Entrepreneurship Education (in press).


References (2)

1. Mayer, I. S., van Dierendonck, D., van Ruijven, T., & Wenzler, I. (2013). Stealth Assessment of Teams in a Digital Game

Environment. In GALA 2013 Conference, Paris (pp. 1–13). Paris, France: Springer.

2. Mayer, I. S., Warmelink, H. J. G., & Bekebrede, G. (2013). Learning in a game-based virtual environment: a comparative

evaluation in higher education. European Journal of Engineering Education, 38(1), 85–106.

doi:10.1080/03043797.2012.742872

3. Mayer, I. S., Warmelink, H. J. G., & Zhou, Q. (2014). The Utility of Games for Society, Business and Politics: A Frame

Reflective Analysis. In Nick Rushby & D. Surry (Eds.), Wiley Handbook of Learning Technology (in press). Wiley.

4. Mayer, I. S., Wolff, A., & Wenzler, I. (2013). Learning Efficacy of the “Hazard Recognition” Serious Game: A Quasi

Experimental Study. In M. Ma, M. F. Oliveira, S. Petersen, & J. Baalsrud Hauge (Eds.), 4th International Conference, SGDA

2013, Trondheim, Norway, September 25-27, 2013. Proceedings (pp. 118–129). Berlin Heidelberg: Springer Verlag.

doi:10.1007/978-3-642-40790-1_12


Review articles

1. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical

education: A literature review. Computers & Education, 53(3), 603–622. doi:10.1016/j.compedu.2009.04.001

2. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of

empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686.

doi:10.1016/j.compedu.2012.03.004

3. Vogel, J. J., Vogel, D. S., Cannon-Bowers, J. A., Bowers, C., Muse, K., & Wright, M. (2006). Computer Gaming

and Interactive Simulations for Learning: a Meta-Analysis. Journal of Educational Computing Research, 34(3),

229–243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM

4. Gosen, J., & Washbush, J. (2004). A Review of Scholarship on Assessing Experiential Learning Effectiveness.

Simulation & Gaming, 35(2), 270–293. doi:10.1177/1046878104263544


Towards a Science of SG (SoSG)?

Frames and discourses Methodology Research designs

and data-gathering

Validated research instruments and tools

A dynamic body of knowledge identifying

the state of the art and knowledge gaps.

Professional ethics of the SG designer, the SG advocate, the SG

seller, the SG interventionist, etc.


Game theory versus gaming

Game theory Gaming

Rigid rule-based, closed Free form, seminar, open

Formal, mathematical, quantifiable, economics,

psychology

Informal, social, interpretative and qualitative,

social, political intervention sciences

Experimental control, objective, separation

researcher and subjects, large ‘ n’ , player cannot

change the rules or setting

Semi- or non experimental, subjective, interaction

between researcher and subjects, small ‘ n’ ;

players should change the rules or setting

Theory-based, hypothesis testing, generalization,

prediction and forecasting

Theory construction, exploratory, hypothesis

formulation, constructing the future

‘ Players’ are research objects;‘Players’ are learning subjects; they bring with

them tacit knowledge, social relations etc.


Framing game related research

Frame Example

Research theory Game theory as in economics, political science

Research concept Political decision-making as a strategic game

Research object Studying game cultures, game economics, game

politics

Design artifact Socio-technical design etc.

Research method Quasi-experiment

Intervention method Therapy, learning, change or decision-method

Data-gathering method Observation, group interview, data-modeling


Strong/weak aspects of gaming as a research method

Strong aspects Weak aspects

Flexible, adaptable, multi-purpose,

complementary

Time and resource consuming. Difficult to

manage. No guarantee for success.

Multi-, interdisciplinary, innovative, appealing Problematic legitimization as ‘science’

Research in action, interaction with real

stakeholder, problem owners, users, etc.

Dependency on others (financers, players,

stakeholders, designers).

Deliverable by or end product. Game takes over the research

Concrete, practical, tangible, fun A lot of fuzz that is not research

… …


Requirements for SG Meth.

Requirements Meth. SG Research

Broad in scope

Comparative

Standardized

Specific

Flexible

Triangulated

Multi-leveled

Validated

Expandable

Unobtrusive

Fast and non-time

consuming

Multi-purpose

d


Underlying research questions

•(How) do (virtual) gaming experiences (a)(e)ffect learning of professionals in socio-technical systems / multi-actor contexts?

•(How) does (virtual) game-based learning (a)(e)ffect Real World (RW) policy-making in socio-technical systems / multi-actor contexts?

What is the relation between (virtual) gaming and (professional)

learning in socio-technical systems / multi-actor contexts?

•Development and testing of design and validation theories, methods & tools•Validation studies of specific and generic game-based artifacts and events.

Design-oriented research (artifact):

‘making it (better)’

•Development and testing of game evaluation, measurement and feedback theories, methods & tools

•Studies into the learning effectiveness of game-based interventions•Studies into the transfer game-based interventions to the RW

Policy, management oriented research (intervention):

‘making it work’

•Studies into the design and use of SG as a research instrument, e.g. e.g. quasi experimental, questionnaires, video observation, Q-method, etc.

•Studies using game based research in infrastructures.

Domain-oriented research (energy, rail, water, ports, tunnels etc.):

‘making it matter’

•Studies on SG in a cultural, organizational, political, economic context•Theory construction on serious gaming ‘as’ / ‘in’ socio-technical / multi-actor systems

Scientific game research:

‘making it understandable’


Serious gaming

Case study

Computer simulation

Experimental research

Participatory research methods

Interviews

Surveys

Document

analysisGroup interviews focus groups

Lage schale data

analysis

Scenario analysis

Panel research

Quasi experim

ental research

Bio & psycho-Metric tests

Q methodo

logy

Videoobservat

ion

(participatory)

Observation

Eye tracking

Crowd sourcing

Path tracking

Mayer, I. S., Warmelink, H., & Bekebrede, G. (n.d.). Learning in a Game-based Virtual Environment: a Comparative Evaluation in Higher Education. European Journal of Engineering Education.


Shortcomings

Few indications how to use the models, for what purpose, with what scope and under what conditions.

Few procedures how to validate the conceptual research / evaluation model.

Few research hypothesis and research designs.

Few definitions, relations and interrelations between the concepts in the model.

Few operationalization and validation of constructs.

Furthermore in the application of the models we see:

A dominance of single case studies, one game, one context of application.

Lacking information on the questionnaires used.

A focus on GBL of children in formal education; little attention to advanced-professional learning, outside education;

A focus on learning of individuals in formal training or educational context; little attention to learning of teams, groups, organizations, networks or systems in policy or organizational context;


Research framework

2. Theory: Complex, multi actor systems and policy

making.

3. Method: Modeling, simulation & gaming

(MSG)

1. Domain application: infrastructures

1.1 Water, Rail, Ports, Energy, Tunnels, etc...

FramingFraming

Sub question 4

sub

que

stio

n 6

Research question 2

Research question 1

Research question 3

Sub question 5

2.1 Complex adaptive systems, resilience, integrated planning,

self-organization, sense-making, etc...

3.1 Design and evaluation of (Serious)

Gaming-Simulation

Fram

ing

Case-Experiments,

e.g. Levee Patroller, SimPort, Water Game,


Research designMSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP

When? Pre-game In-game Post-game

Observation number

O1

O2 O3 O4 O5 O6

How? Online surveyPaper

quest.Paper

quest.Paper

quest.End of game debriefing

Online survey

What? Soc. Dem. Analysis of

maps

Involvement in

MSPMSP

processMSP

processInfluence

After action review

Knowledge in

MSP

Influence in MSP Game play

MSP in country Emotions

Valid response 63 50 40 41 41 38

Additional data gathering

Video registration – Observation – Data logging


Quasi experimental design(Mayer, et al. under review EJEE, BJET, 2012)

XSerious Gaming session

O3

Time

In game data logging

Obse

rvati

on

O4

Learning proces

O2

Inte

rventi

on p

roce

ss

Tra

nsfe

r pro

cess

O1 = Observation 1 Learning objectives, contextO1 = Observation 2 pre-game measurementX1 = Intervention = gaming sessionO3 = Observation 3 post game measurementOx1...n = In game observations / measurementsO4 = Observation 4 Learning objectives, context

O1 Case study

Ox1 Ox3Ox2 Oxn


Measuring indicatorsMSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP

Quantitative Qualitative

Self-reported by the participants

The level of engagement of the players

in the game.

The influence attributed to each

stakeholder, by other stakeholders.

The quality of each of the four MSPs as

assessed by each stakeholder on

ten criteria.

The lessons and insights on the

process and outcome reported

during and after the game.

Observed by the facilitators

The square nautical miles assigned to

the different spatial functions in the

four MSPs.

The observations on how the MSP

process goes, by the game

facilitators.

The quality of each of the four MSPs.

The overlap, conflicts, internal

contradictions etc. between

different spatial functions in or

between MSPs.


• References

• Mayer, I. S., Warmelink, H., & Bekebrede,

G. (n.d.). Learning in a Game-based Virtual

Environment: a Comparative Evaluation in

Higher Education. European Journal of

Engineering Education.

• Mayer, I. S. (2012). Towards a

Comprehensive Methodology for the

Research and Evaluation of Serious

Games. Vs games (pp. 1–15). Genoa:

Procedia Computer Science 00 (2012) 000–

000. Retrieved from

www.sciencedirect.com


EXPKNOW_PRE

SOCSKILLS_ PRE

PROFSKILLS_ PRE

Learning expectations

AGE

SEX

ACHIEVEMENT

Learning style

preferences LRNPREF

Attitude VLEVALUEDL

Attitude GBL EXPSERGAME_ PRE

Engagement FUN

- .12

.2 (t-test)

.12*

.18

.2

.42

0.11

.24

.17

.25

.28*

.40

.46

.14

.42

FREQ. DIGITAL GAMES

FREQ NON-DIGITAL GAMES

FREQ SG

1.1 (t-test)

Attitude GBL EXPSERGAME_POST

EXPKNOW_POST

SOCSKILLS_POST

PROFSKILLS_ POST

Learning satisfaction

MOTIVATION

.22

.22

.19

.34

.24

.1

.24

.13

.16*

.17*

.18*

.17*

.27

.26

.39

.27

.39

-.17

.16

.12*

.3

.1

.62

.34

.27

.18

.32

.46

.53

.58

.78

Quality VLE CYBFUNCTION;

CYNCLEARNESS; CYBACCESS; CYBIMPRESS; CYBCONTENT

Quality game design

QUAGAME; RELGAME

Quality facilitator QUAFACIL.

Game play QUAPLAY;

EFFORTPLAY

Q game

Would like to play other SGs in education.

.2*

.45

.36

.41

.43

.26

.28

.51

.38

.44

.39

.38

.34

.48

Structural equation modelling (Mayer, et al. under review EJEE, 2012)