Upload
michael-burnett
View
261
Download
0
Embed Size (px)
DESCRIPTION
Presentation to British Psychological Society Division of Occupational Psychologists, 2011
Citation preview
Representing Situations in Assessment –
extracting better value from our investment
Division of Occupational Psychology
Annual Conference 2011
Michael Burnett & Almuth McDowall
• The Problem with Situations
• Disentangling Situational Influences:
• Part 1: Predominance of situational effects in high stakes
Assessment Centres
• Part 2: Understanding trait v. situational effects in low stakes
Situational Judgement Tests
• Part 3: How cognitive Situational Models address sources of
situational effects
• Implications
Overview
•Techniques used today range from ‘context free’ cognitive ability
and personality measures to simulations (ACs and SJTs) of work
or organisational situations
•The first problem is that simulations give significant (ACs .281,
SJTs .18-.382) but in absolute terms low additional incremental
validity over ‘context free’ measures
• The second problem is explaining what simulations actually
measure – e.g. the methods effect in ACs3, construct validity of
SJTs4
• The implication is that applied psychology needs to provide a
better account of individual differences in dealing with situations *
1AC Criterion Validity –Harmelin et. al. (2007) 2SJT Criterion Validity - Christian et. al. (2010) 3Sackett & Dreher (1982) 4McDaniel & Whetzel (2005)
The Problem with Situations
Multimedia based simulations will happen
• Multimedia-based ‘simulations’ in assessment will grow quickly
• Video-based, Avatar-based simulations (Second LifeTM), 2-D
Animations, Apps (i-phone/ipad/mobile), Games
• Occ. Psych. Design Methodology needs to adapt
• Maintain control over what is being measured!
Model x2 df GFI AGFI RMSEA NNFI SRMR
Study 1a
Traits Only
(CT0E)
3034.6 67 0.69 0.51 0.22 0.53 0.17
Exercises Only
(0TCE)
96.6 57 0.99 0.97 0.027 0.99 0.014
Correlated Traits
& Exercises
(CTCE)
208.05 59 0.97 0.95 0.052 0.98 0.047
Correlated
Exercises &
Traits (CECT)*
62.45 44 0.99 0.98 0.021 1.0 0.013
Exercises Measures
T
r
a
i
t
s
Part 1: The predominance
of the situation
Structural Equation Models of assessment centre results based on
situations (exercises) provide a good fit to the data, trait-only
(competency dimensions) models do not fit
The best fit combines situations moderated by trait (competency
dimensions) factors
Adding in Cognitive ability and Big 5 Personality dimensions
showed best fit when added to the Exercise model, cognitive ability
predominated
Models that did not fit included Exercise Type model and Trait
Activation model (Lievens et. al., 2006)
Situational effects in high stakes simulations
• Use of low fidelity simulation based on SJT method to explore
comparative effect of situational and ‘trait’ effects
•Academic situations used with samples drawn from University
and job applicants seeking SJT practise (n. 339)
• Situational influence assessed by examining within-situation item
relationships (zero correlation expected if competency/traits
determine performance)
• Competency and personality trait influence assessed by
examining across-situation scale relationships between SJT items
and independent measures of competencies & traits
Part 2: Exploring situational effects
Item 1 Item 2 Item 3 Item 4
Plan &
Organise
Relate &
Network Analysis
Persuade &
Influence
Conscientiousness Openness Agreeableness Extraversion
Item 1 Item 2 Item 3 Item 4
Situation
1
Situation
2
Design of Situational Judgement Items
You have been selected to present a lecture to your course in
front of the head of Department and senior tutors
1. You set yourself objectives for what you are going to say to
include new and innovative ideas (Planning_Open)
2. You try and make sure that the content of your lecture will appeal
to both students and staff (Relating_Agreeable)
3. You aim to keep the audience captivated by questioning
established assumptions (Analysis_Extravert)
4. You check your material and rehearse the presentation to ensure
you appear credible (Persuading_Conscientious)
•
Situational Judgement Item
Plan &
Organise
Relate &
Network Analysis
Persuade &
Influence
Conscientiousness Openness Agreeableness Extraversion
Plan &
Organise
Relate &
Network Analysis
Persuade &
Influence
Conscientiousness Openness Agreeableness Extraversion
.36 .33 .24 .10
.22 .31 .05 .13
Likert what I ‘would do’ responses
Average
Situation .12
(.06 to.20)
• Situations are multidimensional - consider effects on encoding & retrieval:
(Zwaan & Radvansky, 1998; Magliano et.al. 2008):
- Space – context, layout and movement
- Time – chronology of events, temporal shifts
- Protagonists - Actors & Objects
- Intention – Goals & Motivations
- Causation – inferences of what has caused events or what could
happen next
•
Part 3: Understanding situational effects
Goals 1. Space
2. Time
Manager3
Participant
as
Protagonist3
Supplier3
Goals & motivations4
Goals & motivations4
Goals & motivations4
Assumed ‘agreement’ becomes disagreement
‘Non-performing’ supplier becomes threat to supply chain
Manager’s expectation becomes “use of initiative”
Causality5
Assumption
about power
relationship5
Assumption
about power
relationship5
The Situational Judgement determines these…………..
Cognitive Model of Situation
3. Actors 4. Motivations 5. Causes
Goals Space
Time
Tutor
Protagonist
Student
Goals & Motivations
Goals & Motivations
Goals & Motivations
Influence on future evaluation
Influence on future relationships
A
B
C
D
E
F
G
Objective
characteristics
of Situation
a
b
c
d
e
f
g
Situation
as construed
& encoded
I
II
III
IV
V
VI
VII
Potential
behaviours
afforded
i
ii
iii
iv
v
vi
vii
Actual
behaviour
enacted
Adapted from Reis, 2008
Person Variables e.g. goals, domain knowledge
competences, personality traits, state
Personality & Competency effects on
encoding & judgements
•Performance in Simulation-based assessments is determined by
both situational and dispositional (trait & competency) factors
•AC feedback should reflect both aspects
•SJTs can be structured to return required facets
•Situational Models hold promise in improving the yield from
simulations (if we change the paradigm)
•assessing encoding as well as judgement to widen the range
of information gathered from assessments
•providing a systematic way of describing situations to
understand how these affect behaviour (IRT correlates)
•enabling greater control over how situations are used in
simulations including use of multimedia & gaming techniques
(impact on Occ Psych methodology)
Implications