Assessment Center: Evaluation of a Developmental Assessment
Center in Veterans HealthcareCenter in Veterans Healthcare
Boris Yanovsky, Robin L. Graff-Reed, Kasey Kruer, Katerine Osatuke, and Sue R. Dyrenforth
Traditional Vs. Developmental
• Assessment Centers (ACs) are popular tools for selection and promotionselection and promotion
• Also used for developmental purposes – Developmental Assessment Centers (DAC)( )
• These two types are very different from each other:• Prediction vs Development
• DAC used by Veterans Health Administration within a leadership development program for aspiring executivesexecutives
• We were interested to see to what extent our DAC was indeed developmental
What Did We Examine?
• Differences in scoring between ratersDifferences in scoring between raters (Mentors and Preceptors)
• Perceived learning that occurred across i i iactivities• Developmental properties of dimensions (VHA
competencies targeted by the DAC)competencies targeted by the DAC)
• Construct validity of dimensions• Construct validity of dimensions
VHA Core Competencies1. Interpersonal Effectiveness2 Customer Service2. Customer Service3. Systems Thinking4. Flexibility/Adaptability5. Creative Thinkingg6. Organizational Stewardship7 Personal Mastery7. Personal Mastery8. Technical Skills
Assessment Center Exercises Used to Develop Competencieseve op Co pe e c es
1. Meeting Management• Interpersonal Effectiveness, System Thinking, and
Technical Skills
2 Performance Based Interview2. Performance Based Interview• All eight competencies are assessed
3. Executive Team Simulation3. Executive Team Simulation• All eight competencies are assessed
4. Press Conference Simulation4. Press Conference Simulation• All but Technical Skills and Creative Thinking
5. Inbox Simulation• Not aligned with Core Competencies
Executive Team Simulation Rater Differences: All Three Dayse e ces: ee ys
5
3
43.
96 4.00
3.81
3.96
3.96
4.01
3.99
4.03
4.05
4.10
3.94
4.04
3.94
3.87
3.88
4.01
2
3
Mentor
1Preceptor
0ctiveness
Thinking
l Mastery
ical Skills
ardship
daptability
er Service
Thinking
Interpersonal Effectiv
Systems Th
Personal MTechnica
Organizational Stewar
Flexibility/Adapt
Customer
Creative Th
Executive Team Simulation Rater Differences: Mondaye e ces: o d y
5
3
43.
733.
89
3.58
3.89
3.88
3.90
3.81
3.98
3.9
4.07
.75
4.04
.73
3.80
3.76
4.01
2
3
Mentor
3 3 3 3 3 3
1Preceptor
0ctiveness
Thinking
l Mastery
ical Skills
ardship
daptability
er Service
Thinking
Interpersonal Effectiv
Systems Th
Personal MTechnica
Organizational Stewar
Flexibility/Adapt
Customer
Creative Th
Executive Team Simulation Rater Differences: Tuesdaye e ces: uesd y
5
3
43.
95 3.99
3.8 3.
87
3.83
3.96
3.95
4.07
3.94
3.99
3.95
3.87
3.9
3.68
3.8
3.91
2
3
Mentor
3
1Preceptor
0ctiveness
Thinking
l Mastery
ical Skills
ardship
daptability
er Service
Thinking
Interpersonal Effectiv
Systems Th
Personal MTechnica
Organizational Stewar
Flexibility/Adapt
Customer
Creative Th
Executive Team Simulation Rater Differences: Thursdaye e ces: u sd y
5
1 6 0 9 .25
43
44.
31 4.20
4.17
4.21
4.27
4.26
4.3 4.
09
4.43
4.30
4.22
4.2
4.31
4.
4.21 4.
14
2
3
Mentor
1Preceptor
0ctiveness
Thinking
l Mastery
ical Skills
ardship
daptability
er Service
Thinking
Interpersonal Effectiv
Systems Th
Personal MTechnica
Organizational Stewar
Flexibility/Adapt
Customer
Creative Th
Press Conference Rater Differences
5
3
43.
6 3.65
3.69 3.
69
3.81
3.84
3.94
4.08
2
3
Mentor
3 3
1
2Preceptor
0king ing
kills ure
Systems Thinking
Ethical Reasoning
Listening Skills
Maintains Composure
Construct Validity of Dimensions: yDimensions Within Exercises Do
Not Differentiate From Each OtherNot Differentiate From Each Other
ETS All 3 Days with Press Conference
Pres
s
Pres
s
ess C
onf
Pres
s
ETS
ETS ET
S
v. in
ETS
ty in
ETS
ETS
tem
s Thin
king
in P
Conf
ical R
easo
ning
in Co
nfen
ing S
kills
in Pr
ent
ains C
omp.
in P
Conf
tem
s Thin
king
in E
Stew
ards
hip in
Esto
mer
Ser
vice
in
r-per
sona
l Effe
ctiv
xibilit
y / A
dapt
abilit
yso
nal M
aste
ry in
Syste
Ethic
Liste
Main
Syste
Org S
Cust
Inte
r
Flex
i
Pers
Systems Thinking in Press Conf Pearson r XEthical Reasoning in Press Conf Pearson r .569(**) XListening Skills in Press Conf Pearson r .670(**) .581(**) XMaintains Comp. in Press Conf Pearson r .684(**) .438(**) .732(**) XSystems Thinking in ETS Pearson r .415(**) .323(*) .438(**) .279(*) XOrg Stewardship in ETS Pearson r .451(**) .423(**) .320(*) 0.238 .622(**) XCustomer Service in ETS Pearson r .460(**) .420(**) .373(**) .372(**) .573(**) .626(**) XInter-personal Effectiv. in ETS Pearson r .456(**) .455(**) .481(**) .437(**) .545(**) .632(**) .616(**) XFlexibility / Adaptability in ETS Pearson r .515(**) .362(**) .469(**) .504(**) .629(**) .696(**) .656(**) .711(**) XPersonal Mastery in ETS Pearson r .421(**) .300(*) .385(**) .375(**) .449(**) .637(**) .576(**) .738(**) .617(**) X**. Correlation is significant at the 0.01 level (2-tailed).
* C l ti i i ifi t t th 0 05 l l (2 t il d)
Sample size fluctuated between 57-61
*. Correlation is significant at the 0.05 level (2-tailed).
ETS Monday with Press Conference
ss C
onf
Pres
s
ss C
onf
ress
ETS
TS ETS in E
TS
y in
ETS
ETS
tem
s Thin
king
in Pr
esica
l Rea
soni
ng in
PCo
nfte
ning
Skil
ls in
Pres
inta
ins C
omp.
in P
reCo
nfte
ms T
hink
ing
in ET
Stew
ards
hip
in E
TSst
omer
Ser
vice
in E
r-per
sona
l Effe
ctiv.
xibilit
y / A
dapt
abilit
y so
nal M
aste
ry in
ET
Syste
Ethic
Liste
Mai
n
Syst
Org
Cust
Inte
r
Flex
Pers
Systems Thinking in Press Conf Pearson r XEthical Reasoning in Press Conf Pearson r .569(**) XListening Skills in Press Conf Pearson r .670(**) .581(**) XMaintains Comp in Press Conf Pearson r 684(**) 438(**) 732(**) XMaintains Comp. in Press Conf Pearson r .684(**) .438(**) .732(**) XSystems Thinking in ETS Pearson r 0.247 0.126 0.248 0.084 XOrg Stewardship in ETS Pearson r .297(*) .302(*) 0.136 0.009 .616(**) XCustomer Service in ETS Pearson r .261(*) .347(**) 0.249 0.106 .549(**) .694(**) XInter-personal Effectiv. in ETS Pearson r 0.233 0.190 .279(*) 0.177 .560(**) .614(**) .639(**) XFlexibility / Adaptability in ETS Pearson r .265(*) 0.241 .297(*) .276(*) .646(**) .624(**) .549(**) .727(**) XPersonal Mastery in ETS Pearson r 0.163 0.145 0.152 0.089 .497(**) .600(**) .593(**) .722(**) .692(**) X
Sample size fluctuated between 56-61
**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).
Sample size fluctuated between 56 61
ETS All 3 Days with Meeting Managementge e
Mgm
t
Mgm
t
n M
tngM
gmt
hinkin
g in
Mtn
g M
Skills
in M
tng
Mg
vene
ss i
n ET
STh
inking
in E
TSSk
ills in
ETS
I/P S
kills
in
Syst
em T
h
Tech
nica
l S
I/P E
ffect
iv
Syst
ems T
Tech
nica
l S
I/P Skills in MtngMgmt Pearson r XSystem Thinking in Mtng Mgmt Pearson r 856(**) XSystem Thinking in Mtng Mgmt Pearson r .856(**) XTechnical Skills in Mtng Mgmt Pearson r .735(**) .836(**) XI/P Effectiveness in ETS Pearson r -0.072 0.097 0.154 XSystems Thinking in ETS Pearson r -0.156 -0.037 0.109 .545(**) X
S PTechnical Skills in ETS Pearson r -0.146 0.048 0.125 .599(**) .651(**) X
N=61**. Correlation is significant at the 0.01 level (2-tailed).
Meeting Management with Press ConferenceCo e e ce
Conf
onf
Mgm
t
nkin
g in
Pre
e Co
kills
in P
ress
Co
king
in M
tng
Mg
alSk
ills in
Mtn
g
Syst
ems
Thin
List
enin
g Sk
ilSy
stem
Thi
nkIn
terp
erso
nal
Systems Thinking in Press Conf Pearson r XSystems Thinking in Press Conf Pearson r XListening Skills in Press Conf Pearson r .670(**) XSystem Thinking in Mtng Mgmt Pearson r 0.170 0.177 XInterpersonal Skills in Mtng Pearson r 0.155 0.112 .856(**) X( )
Sample size fluctuated between 58-61
**. Correlation is significant at the 0.01 level (2-tailed).
Developmental Properties of Dimensions: Scores Increased
Across AdministrationsAcross Administrations
ETS Scores Across 3 Days5
3
4
2
3
MondayTuesday
1
TuesdayThursday
0ctiveness
s Thinking
al Mastery
nical Skills
wardship
daptability
er Service
e Thinking
Interpersonal Effecti
Systems T
Personal MTechnic
Organizational Stewar
Flexibility/Adap
Customer
Creative T
Inbox and PBI Scores Across 2 Days
1 5
0.8
0.9
4
0 5
0.6
0.7
Day 1
3
Day 1
0.3
0.4
0.5 Day 1Day 2
2
Day 1Day 2
0
0.1
0.2 1
0Inbox 0
PBI
Summary• Differences Between Raters: Mainly non-
significant• Raters (Mentors and Preceptors) tended to agree
with each other on ratees’ (Candidates’) ratings• Perceived Learning/DevelopmentPerceived Learning/Development
• Participants improved on scores in activities with multiple administrations, showing evidence for the d l t l t f th t t d t idevelopmental nature of the targeted competencies
• Construct Validity• Competencies did not measure separate constructsCompetencies did not measure separate constructs• This finding on our DAC is consistent with previous
research on DACs
Future Considerations
• Further training using behavioral anchors for raters
• Would allow for raters to better diff ti t b t di idifferentiate between dimensions
• Increases construct validitySi ll C did t l t d b• Since all Candidates were evaluated by separate Raters, true inter-rater reliability could not be assessedreliability could not be assessed
• Techniques exist for breaking down variance and could be used for future DACvariance and could be used for future DAC evaluation