Dr. Christof Obermann Jarosław Chybicki Assessment Center around the world: Where do we stand? What...

Dr. Christof Obermann

Jarosław Chybicki

Assessment Center around the world: Where do we stand?

What is new?

1. Assessment Center – how common is this method?Look on the usage around the worldSpecial: Empirical survey of AC application in

Germany2. How valid are Assessment Center?

Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

Importance of personal networks

+ France, ItalyLegal regulations, labor

market Workers Council

+ Germany, Italy, France- Switzerland

Tradition of psychometric

testing- Germany

- + Great Britain

Openness of the culture / exports

+ Netherlands, Northern Europe,Switzerland

AC-Use

Different weight of criteriae.g. Eastern Europe /

Southern Europe

AC use in European countries

Abb. 1.4-8

German-speaking countries USA

Group discussion

In-Basket Interview Role play Presentation Fact-finding. case study

´Testing0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Sources: Arbeitskreis AC, 2001 Spychalski et al. , 1997

Type of exercises: Europe vs. USA

4

3,7

3,7

3,5

3,4

3,4

3,3

3,1

3,1

3

3

2,9

2,9

2,8

2,7

2,6

2,6

2,5

2,3

1,7

1,6

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5

Malaysia

Schw eden

Belgien

Neuseeland

Südafrika

Großbritannien

Portugal

Spanien

alle Länder

Niederlande

Irland

Hong Kong

Kanada

Australien

Griechenland

Singapur

Frankreich

USA

Japan

Deutschland

Italien

Einsatzhäufigkeit

Usage of psychometric testing

1. Assessment Center – how common is this method?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

73,4%

26,6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Yes No

companies and AC

Most of listed companies uses AC

Be

gin

nin

g o

f A

C

Number

Use of AC-method since …

Frequency of the AC

study 2007

study 2001

No information

Increasing Constant Declining

Is AC rising or declining?

AC runs in one Year

study 2007

study 2001

How many runs by company?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Personalauswahl Entwicklungsmaßnahme Potenzialanalyse

AC Studie 2008 AC Studie 2001

Objectives of AC?

Yes No No Information

Accomplishment necessary for next steps

study 2007

study 2001

Other

Biographical Questionnaire

Group Discussion with role players

Computer – based model

Fact Finding

Role play

Personality test

Cognitive testing

In Basket

Group Discussion with candidates

Interview

Case study

One-on-one Interview

Presentation

What exercises / elements used?

study 2001

study 2007

Internal psychologist

Coach

Board member

Experts

Other:

Agent

Workers council

Manager

Consultants

HR

Who are the assessors?

Average number of participants in a AC

study 2007

study 2001

131 2-4 5-7 8-10 11-13

How many participants attending?

study 2007

study 2001

On objectives

On dimensions

None Behavior expected

Information for candidates provided?

Feedback

Verbal and in written

form

Verbal In written form

Neither

study 2007

study 2001

How does the participant receive feedback?

Just after Assessment

Up to one week later

More than one week

later

No information

study 2007

study 2001

When is the feedback given?

Up to 15 15-30 30-45 45-60 60-90 more than 90 minutes

Feedback in minutes

study 2007

study 2001

How long does the feedback take?

Up to 1 year 1 – 2 3 – 4 more than 4

study 2007

study 2001

Second chance?

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

• Knows (and develops) own strengths and weaknesses• Ability to learn

Motivation and ability to learn

• Develops new ideas / innovations / inventions which could be named as “out of the box”

• Open for new ideas

Openness

• Physical fitness• Copes with pressure, high expectations and complexity while

maintaining performance level

Toughness and stress management

• Demands from others´ only what he / she is willing to do him- / herself

• Acts as an ambassador for important values and aspirations• Is open about personal beliefs and feelings

Extraversion / social competency

• Shows commitment, determination and enthusiasm in delivering required business results

• Developing of (own) ambitious goals (driving for excellence)

Drive and Commitment

• Expands the big picture by detecting long-term and side-effects or new interdependencies

• Explains complex subjects in simple words and abstracts the relevant messages

• Above average grasp of new subjects

Handle complexity

Observable behaviorPotential criteria

Future leaders - what is expected in general?

M a n a g e m e n t - P r o g r e s s S t u d i e v o n A T & T

A b b . 3 . 2 - 5

P r ä d i k t o r

K a n d i d a t w i r d i n a c h tJ a h r e n i m m i t t l e r e nM a n a g e m e n t s e i n

K r i t e r i u m

N a c h a c h t J a h r e nt a t s ä c h l i c h i m m i t t l e r e nM a n a g e m e n t

N M i t C o l l e g e O h n e C o l l e g eJ a 2 0 3 6 4 % 4 0 %N e i n / f r a g l i c h 2 1 9 3 2 % 9 %

V a l i d i t ä t s k o e f f i z i e n t . 4 6 . 4 6

N a c h T h o r n t o n & B y h a m , 1 9 8 2

1948 – Historical management progress study

Validity measures of diagnostic tools

1998 – Metaanalysis

Validity Additive validity

Additive percentage on validity

Standardized regression

Intelligence test .51 Work sample .54 .12 24% .41 Integrity test .41 .14 27% .41 Interview - structured .51 .14 27% .41 Interview – not structured .31 .07 14% .27 Test on technical knowleged .48 .07 14% .31 First weeks in the job .44 .07 14% .31 Peer rating .49 .07 14´% .31 Job reference letters .26 .06 12% .26 Assessment center .36 .01 2% .14 Graphology .02 .00 0% .02

2003 - Validity of Assessment Center by dimension

Meta-analysis: Regression of the AC-dimensions

15 Added dimension R R² R²

1 Problem solving .39 .15

2 Influencing others .43 .18 .03

3 Organizing and planning .44 .19 .01

4 Communication .45 .20 .01

5 Drive .45 .20 .00

6 Consideration/Awarness of others .45 .20 .00

Gradual regression, starting with highest validity

Source: Arthur, Day & McNelly (2003)

p SDpUntere Grenze

Obere Grenze

Gesamt 11.136 106 40 .22 .14 .26 .13 -.01 .52

Leistung 4.198 49 29 .20 .08 .28 .00 .28 .28

Training 3.503 15 10 .31 .10 .35 .09 .16 .53

Absatz 267 15 4 .11 .05 .15 .00 .15 .15

Beförderung 1.738 13 10 .27 .15 _d _d -.02 .56

Fluktuation 1.430 9 6 .07 .15 _d _d -.22 .36

Anmerkungen: a Anzahl der unterschiedlichen Validitäten aus unterschiedlichen Stichproben. Dies ist die Anzahl der Validitäten, auf der die Berechnungen basieren. b Für Beförderung und Fluktuation waren keine Korrekturen bezüglich der Kriteriumunreliabilität möglich

Anzahl der

Studien

(k)a

Anzahlderr's

Stich-proben-größe

Korrigiertbzgl.

Kriteriums-unreliabiltät

90%-Vertrauens-intervalle

SDr

mittleresr

Hardison, C.M. & Sacket, P.R (2007)

1. Enrollment and clarification of roles

2. Job analysis

3. Development of exercises

4. Observation and rating

5. Selection of observers and training

6. Pre-selection and preparation

7. Organization and execution

8. Feedback and follow-up

9. Evaluation

http://www.arbeitskreis-ac.de/

German Quality Standards on AC

Pitfalls:Simple sampling of criteriaJob analysis only looking to the pastBiased methods which lead only to simulation or testing

Job analysis

Pitfalls:Use of ethically difficult exercises like NASA exerciseFocus on group discussions for economic reasons“If you were ...“ simulationsInstructions with assignments for behavior

Exercises

Pitfalls:No operationalizing of criteriaFinal assessment without pre-defined rules

Ratings

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

The more content the more valid …

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

Protocol vs. guided assessment„Protocol assessment“ „Guided assessment“

Kundenorientierung

Customer orientation

Conflict ability

Argumentation

notes notes

Different types of observation tools

Observation sheet 1: Type: polarity profile - unspecific appellation of poles

Dimension: Conflict ability

Definition: Does not avoid conflicts, tries to find solutions, includes oppinion of others

Polarity profile

1 2 3 4 5

Has to concentrate on one single project

Is able to handle more than one project at the same time

Wird mit vagen Projektlinien fertig, sofern sie in konkreten Strukturen oder Methoden verankert sind

Kann mit nichtspezifizierten Zielvorhaben und unter Unsicherheit arbeiten, sofern gelegentlich Rückgriff auf einen konkreten Orientierungspunkt möglich ist

Tolerates ambiguity in projects and is able to handle long-term projects

Observation sheet 2: Type: polarity profile - unspecific appellation of poles

Did

not

sho

w t

he o

bser

ved

beha

vior

Did

not

mat

ch t

he n

eede

d cr

iter

ia

Unt

erdu

rchs

chni

ttli

ch:

erfü

llt

im A

llge

mei

nen

nich

t di

e er

ford

erli

chen

V

erha

lten

skri

teri

en -

qu

anti

tati

v un

d qu

alit

ativ

Dur

chsc

hnit

tlic

h:

ents

pric

ht d

en

Ver

halt

ensk

rite

rien

-

quan

tita

tiv

und

qual

itat

iv

Übe

rdur

chsc

hnit

tlic

h:

über

stei

gt g

ener

ell d

ie

erfo

rder

lich

en

Ver

halt

ensk

rite

rien

-

quan

tita

tiv

und

qual

itat

iv

Ver

y m

uch

abov

e-av

erag

e:

show

muc

h m

ore

than

ne

eded

0 1 2 3 4 5

Comments

Different types of observation tools

Different types of observation tools

Observation sheet 3:Type: behavioral checklist

Dimension: Decision orientation

Definition: Tries to find solutions, arranges effective checkup actions

Quotes 1 2 3 4 5 - + Structures the conversation (asks questions...)

Analyses, asks questions to find reasons for the problems

Sets clear directives

Structures partial steps, that could solve the problem

Arranges checkup

Overall score

Observation sheet 4:Type: Behavioral coding

Different types of observation tools

Different types of observation tools

Exercise oriented

Observation sheet 5: Type: Exercise oriented

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

Festsetzung eines einheitlichen Standards

Visualisierung der Beurteilungstendenzen/Wahrnehmungsfehler

Höhere Akzeptanz Gesteigerte Effizienz

Schritt 1

Professionelles Aufzeichnen der wichtigsten Übungen auf Video

Schritt 2

Erstellung eines differenzierten Benchmarks

Schritt 3

Beobachtertraining: Darbietung des Videos / Individuelle Beobachtung

Schritt 4

Abgleich Expertenurteil / Beobachterurteil, Reflexion der persönlichen Beurteilungstendenzen

Oberserver training according frame of reference

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

W i e d e r h o l u n g s s i m u l a t i o n i m A C

A b b . 4 . 3 - 1

V o r m i t t a g

M i t a r b e i t e r -R o l l e n s p i e l

B e u r t e i l u n g A

M i t t a g

I n t e r v e n t i o n

N a c h m i t t a g

M i t a r b e i t e r -R o l l e n s p i e l

B e u r t e i l u n g B

P r o b l e m l ö s e k o m p e t e n z

E i n s t e l l u n g e n / W e r t e

K o n z e p t i o n e l l eF ä h i g k e i t e n

A n a l y s e v e r m ö g e n / S t r a t e g i s c h e sD e n k e n

E n t s c h e i d u n g s v e r h a l t e n /H a n d l u n g s o r i e n t i e r u n g

A r b e i t s o r g a n i s a t i o n

F a c h w i s s e n

E i n s t e l l u n g e n u n dW e r t e

L e i s t u n g s m o t i v a t i o n

U n t e r n e h m e r i s c h e s D e n k e n

L e r n b e r e i t s c h a f t

K o m m u n i k a t i v e K o m p e t e n zS o z i a l e K o m p e t e n z K o n t a k t v e r h a l t e n

Ü b e r z e u g u n g s k r a f t

K o o p e r a t i o n s v e r h a l t e n

D u r c h s e t z u n g s v e r m ö g e n

P r o b l e m l ö s e k o m p e t e n z

E i n s t e l l u n g e n / W e r t e

K o n z e p t i o n e l l eF ä h i g k e i t e n

A n a l y s e v e r m ö g e n / S t r a t e g i s c h e sD e n k e n

E n t s c h e i d u n g s v e r h a l t e n /H a n d l u n g s o r i e n t i e r u n g

A r b e i t s o r g a n i s a t i o n

F a c h w i s s e n

E i n s t e l l u n g e n u n dW e r t e

L e i s t u n g s m o t i v a t i o n

U n t e r n e h m e r i s c h e s D e n k e n

L e r n b e r e i t s c h a f t

K o m m u n i k a t i v e K o m p e t e n zK o n t a k t v e r h a l t e n

Ü b e r z e u g u n g s k r a f t

K o o p e r a t i o n s v e r h a l t e n

D u r c h s e t z u n g s v e r m ö g e n

D i r e k t e R ü c k m e l d u n g d u r c h B e o b a c h t e r

S e l b s t - E i n s c h ä t z u n g

V i d e o a n a l y s e d e s T e i l n e h m e r s

G r u p p e n t r a i n i n g

Basic idea of learning Assessment

Abb. 4.3-3

Lerngewinne nach Rückmeldung

0

2

4

6

8

10

12

-0,67 -0,33 0,00 0,33 0,67 1,00 1,33 1,67 2,00 2,33 2,67 3,00 3,33 3,67

Anzahl

Quelle: Obermann, 1994

Typcial result of a learning center

1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany

2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards

3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center

4. International AC – look on our projects

Agenda

• 1st leadership level: Teamleader/ Bank Manager

• 3 different parts of the bank: Back-Office, Headquarters, Sales

• Back-Office in Duisburg: e.g. Call Center, Operations, Central Control...

• German Headquarters in Düsseldorf: e.g. HR, Marketing, Project Office...

• Branch Office, Mobile Sales Force, Investment Center

IPAD case study DC for Citibank

1. Day Observer Activity08.00 - 08.30 Introduction for the role players & observer08.30 - 09.00 Welcome!09.30 - 11.00 2x Self-Presentation (leadership understanding) 11.15 - 12.45 2x Case Study (either Back-Office or Sales or Project Management

(Headquarters)13.30 - 15.00 2x Role Play/ Employee Discussion (either Back-Office or Sales or

Headquarters)15.15 - 18.30 4x Role Play/ Team Discussion (either Back-Office or Sales or Headquarters)19.15 - 21.00 Observer Conference

2. Day Observer Activity08.30 - 11.00 2x Interview11.15 - 11.35 Evaluation of the participant‘s self-assessment11.35 - 12.00 Writing the Management Summary12.00 - 14.00 Conference14.00 - 14.30 Debriefing14.30 - 15.00 Individual Feedback for each participant

* 1 HR-Manager + 1 Executive in the observer group, 4 groups of 8 participants

IPAD case study DC for Citibank

Example # 1 - DC Citibank• specific cases & role plays for several units which represent the business

• high commitment to the DC within top management

• high social validity

• high business need to fill the jobs

• high quality of the job-specific criteria

• no psychological tests to measure cognitive skills

• high pressure to be in business, clear processes

• high frustration when the prognosis on position is longer than 6 months

IPAD case study DC for Citibank

Dr. Christof Obermann

Jarosław Chybicki