Adaptive Personality Scales: Initial Validation · PDF fileComputer Adaptive Personality...

1

Computer Adaptive

Personality Scales:

A Military and Civilian Application

Walter C. Borman PDRI, an SHL Company

and University of South Florida

Tracy Kantrowicz

PreVisor, an SHL Company

Outline

• Description of the “adaptive test”

concept

• Development of NCAPS and GPI-

Adaptive

• Initial validation results

• Conclusions and next steps

2

Description of the

“Adaptive Test” Concept

• Initial ability domain application

• Our work began in the performance

domain

• Forced-choice Computer Adaptive

Rating Scales (CARS)

• Laboratory study showed lower

standard error measurement and higher

validity and accuracy for CARS

CARS Performance Estimation

Process

Very Somewhat Very Ineffective Ineffective Effective Effective

First Item Pair 1 2 3 4

Second Pair 1 2 3 4

Third Pair 1 2 3 4

Fourth Pair 1 2 3 4

Final Estimated

Performance 1 2 3 4

3

Development of NCAPS

• Idea was to apply CARS concept to

non-cognitive testing

• 19 potential personality constructs were

identified

• Psychologists rated their importance for

79 Navy jobs

• 10 constructs selected based on means

and SDs

Development of NCAPS (Cont.)

• PDRI generated 1725 items at all trait

levels

• Construct and trait level “retranslation”

was conducted

• 1494 items survived, 106-199 per

construct

4

Example Items for

Social Orientation

• It is easy for me to find something in

common with any person I meet (M =

6.36)

• It takes real effort for me to hide my

impatience with people who aren’t very

bright (M = 1.49)

• I am able to make friends when I put

some effort into it (M = 3.63)

Validation Results

• Concurrent validation study designed

with 110 first tour Navy Sailors

• NCAPS and a conventional personality

inventory administered and supervisor

performance ratings gathered on nine

“Navy-wide” dimensions

• Composite of the ratings was reliable

(interrater r = .53)

5

Validation Results (Cont.)

• Unit-weighted composite of 10 NCAPS

conventional against composite overall

performance (r = .13 uncorrected;

r = .18 corrected)

• Unit-weighted composite of 10 NCAPS

adaptive against composite overall

performance (r = .27 uncorrected;

r = .37 corrected)

Validities Associated With Different Numbers of Item-Pairs

(Dependability)

0.00

0.05

0.10

0.15

0.20

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Item-Pairs

Va

lid

ity

Co

eff

icie

nt

6

Civilian Version:

Global Personality Inventory-Adaptive

• Identified 13 dimensions for item

development

• PDRI and PreVisor generated 2679

items for 13 dimensions

• Construct and trait level retranslation

was again conducted

• 2519 items “survived the cut,” 150-200

per construct

Constructs Identified for GPI-Adaptive

compared to NCAPS Constructs

GPI-Adaptive

• Achievement Striving

• Social Orientation

• Cooperation

• Composure

• Adjusting to Change

• Positive Self-Concept

• Responsibility

• Thoroughness

• Sense of Duty

• Self-Sufficiency

• Learning Orientation

• Directing and Influencing

• Innovation

NCAPS

• Achievement

• Social Orientation

• Stress Tolerance

• Adaptability/Flexibility

• Attention to Detail

• Dependability

• Dutifulness/Integrity

• Self Reliance

• Willingness to Learn

• Vigilance

7

Validation Results

• Concurrent Validity Study Against Supervisor

Performance Ratings (N=1340)

• Unit-Weighted Composite Against Overall

Performance (r=.28, corrected for criterion

unreliability)

• Validities Asymptote at 5-8 Item Pairs

• Testing Time ½ that of Conventional

Inventory