View
6
Download
0
Category
Preview:
Citation preview
Innovations in Gamified Assessment
A GAME CHANGER IN
EMPLOYEE SELECTION
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
“I did my online test yesterday, thought it was quite fun actually. Got way better for the
second part and I think I got 2 or 3 'nailed it!' where I got them all right, but then some I got maybe half only, they kept buzzing out even though I swear they belonged in that spot lol”
From Whirlpool – Graduate Testing Online Forum
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
- Overview of Gamified Assessments
- Value of Gamified Assessments
- What can we measure?
- Case Study Development of Gamified Assessment
Implementation of a Gamified Assessment
Outcomes of using a Gamified Assessment
- Reliability and Validity of Gamified Assessments
Reliability
Validity
- Next Generation of Gamified Assessments
CONTENTS
Engaging Assessments for the Digital Age
OVERVIEW OF GAMIFIED ASSESSMENTS
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Gamified assessments blend game elements with robust psychometric methodologies.
Engaging
Reliable
Valid
Making data gathering more enjoyable
VALUE OF GAMIFIED ASSESSMENTS
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Gamification has consistently toped the list of
upcoming tech trends in HR for the last 5 years.
http://www.forbes.com/sites/jeannemeister/2015/03/30/future-of-work-using-gamification-for-human-resources/#56b5ef4032ba
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Drive to provide a positive
recruitment experience for
candidates.
Leveraging game elements
and narrative to create a
two way interaction.
Why gamify?
WHAT CAN WE MEASURE?
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Leads to increased
intrinsic motivation
Can measure similar
constructs to standard
assessments with
correlations ranging from
rs = 0.45 - 0.60
Performance may be
multidimensional
Lumsden et al (2016) Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy. JMIR Serious Games, 4 (2)
Previous Research on Gamified Assessments
High volume Graduate Recruitment
CASE STUDY
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
- A large professional services firm
- Assesses large volumes (10K+) of
graduates for recruitment purposes
- Create a recruitment solution which
measured cognitive ability
- Positive candidate experience
- Mobile first
- Promote the companies employee
value proposition
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Interactive Gamified Assessments
RECRUITMENT CAMPAIGN INFO
GAMIFIED ASSESSMENT
Interesting Stats So Far VIDEO DEMO
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
1798
4843
3113
338 215
815
1351
Feb Mar Apr May Jun Jul Aug
Completions so far… G
am
ifie
d A
sse
ssm
en
t C
om
ple
tion
s b
y M
onth
12,558
Completions
45 calls to the Helpdesk
(0.35% of total
candidates)
38 Sessions completed
on a mobile device
and 26 on a tablet
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
- Chrome was the most
popular browser with
69% share
- Safari made up 17%
- Firefox 9% and IE 5%
Internet
Explorer
5%
Firefox
9%
Safari
17%
Chrome
69%
Browser Usage
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Example Output
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
REPORT
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Positive Candidate Experience
Whirlpool - Graduate Testing Online Forum
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
CANDIDATE COMMENTS
“The online testing…very cool and different which i
enjoyed”
“The test was pretty enjoyable”
“Had a lot of fun in the testing, prefer this to any other type
of testing.”
“Yep loved this type of test.”
“The test was pretty enjoyable imo. All about logic no
comprehension or math bull lol….”
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
“Talegent has really brought this idea to life. They identified the job-relevant competencies that would translate well to be tested via a game interface and designed the questions and game logic”
Tanyth Lloyd | Talent Acquisition Manager
- Engaging Candidates
with innovative gamified
experience
- Strengthen employment
brand
- Psychometrically sound
Benefits Innovation
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
We matched the gamified campaign
with a traditional recruitment campaign
Graduates
Australian Organisation
Well regarded brand
Found a campaign which shared
31.17% of its candidates with the
gamified recruitment campaign
Gamified
(N =12,763) Traditional (N = 1,745)
Comparing it to traditional assessments
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
The gamified
assessment had lower
drop out rates
compared to the
traditional assessment
campaign.
Drop out Rates vs Traditional Assessments
Traditional Gamified
13.92%
1.18%
Do they measure anything?
RELIABILITY AND VALIDITY
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
0
100
200
300
400
500
600
700
800
900
1000
-555.7 -469.1 -382.5 -295.9 -209.3 -122.7 -36.1 50.5 137.1 223.7 310.3 396.9 483.5 570.1 656.7
Nu
mb
er
of
Ap
plican
ts
Raw Ability Score
Distribution of Gamified Assessment Raw Score
Distribution Of Raw Score
Item Response Theory Model
a
Measuring Ability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 2 3 4 5 6 7 8 9 10
Pro
bab
ilit
y o
f p
lacin
g o
pe
rato
r
co
rre
ctl
y
Learning Experience With Operator
• Previous assessments have been static.
• Not enabling applicants to experiment or learn.
• The gamified assessment provides applicants with this opportunity enabling
the measurement of learning agility.
Measuring Learning Agility
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 2 3 4 5 6 7 8 9 10
Pro
bab
ility
of pla
cing
opera
tor
corr
ect
ly
Operator Usage
Probability of placing operator correctly over operator usage by agility level
High
Average
Low
Looking at the response data we can see that applicants differ in the speed with which they learn the operator function.
Some applicants show high levels of learning agility and learn the dynamic systems quickly, while other applicants take more
experimentation attempts to learn.
Measuring Learning Agility
Dynamic IRT Linked to Learning Agility
The reliability of the Gamified Assessment
“… So gamified assessment with Dynamic IRT are a reliable predictor
of candidate ability.”
And how valid is gamified assessment…?
Gamified Verbal Numeric Logical
Gamified 1.00 0.11* 0.42** 0.41**
Verbal 0.11* 1.00 0.40** -0.02
Numeric 0.42** 0.40** 1.00 0.20*
Logical 0.41** -0.02 0.20* 1.00
** < 0.01, * = 0.05
As expected is weak correlation with Verbal
Reasoning
Verbal Reasoning with r (248) = 0.11, p < 0.05
Moderate correlations with
Numeric Reasoning with r (248) = 0.42, p < 0.01
Logical Reasoning with r (248) = 0.41, p < 0.01
“… and can measure similar abilities to traditional assessments.”
Where to next? NEXT GENERATION OF GAMIFIED ASSESSMENTS
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
New technology is opening up a virtual world to
assessment developers full of amazing new
opportunities to measure abilities and traits in
new and engaging ways. In near future
developers will create high fidelity immersive
simulations of work environments.
These new experiences will provide
opportunities to measure new constructs and
new challenges.
Virtual Reality, Blended
Reality, High Fidelity
Simulations
Gamified Assessments SUMMARY
TALEGENT I n n o v a t i o n s i n G a m i f i e d A s s e s s m e n t
Gamified assessments
can measure similar
abilities to traditional
cognitive ability
assessments
Has lower drop
out rates than
traditional
assessments
Sound reliability
and validity
© 2016 Talegent. All Rights Reserved.
Applying new psychometric assessment
technologies to meet challenges of online
recruitment.
Auckland, New Zealand
Level 18, 48 Emily Place, Auckland City 1010
PO Box 105429 Auckland City 1143
0508 ASSESS (277 377) or (09) 282 4739
Recommended