Designing Assessments of Intrapersonal and

Interpersonal Competencies

Patrick Kyllonen Educational Testing Service

National Academies of Sciences, Engineering, Medicine

Committee on Assessing Intrapersonal and Interpersonal Competencies

Workshop on Assessing Intrapersonal and Interpersonal Competencies

December 14, 2015, 1:35-1:55, National Academy of Sciences Building 2101 Constitution Avenue, NW, Washington, DC 20418

1/14/2016

Today’s agenda

• For the Assessing Competencies Committee on the 14th, please plan to briefly summarize your chapter on designing tests to measure noncognitive skills.

2

Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century

Committee on Defining Deeper Learning and

21st Century Skills

Division of Behavioral and Social Sciences and Education

National Research Council

Study Sponsors

• Carnegie Corporation of New York

• William and Flora Hewlett Foundation

• John D. and Catherine T. MacArthur Foundation

• National Science Foundation

• Nellie Mae Education Foundation

• Pearson Foundation

• Raikes Foundation

• Susan Crown Exchange

• Stupski Foundation

Committee

• JAMES W. PELLEGRINO (Chair), Learning Sciences Research Institute, University of Illinois at Chicago

• GREG J. DUNCAN, Department of Education, University of California, Irvine

• JOAN L. HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles

• MARGARET A. HONEY, New York Hall of Science, Queens, New York

• PATRICK C. KYLLONEN, Center for New Constructs, Educational Testing Service

• HENRY M. LEVIN, Teachers College, Columbia University

• CHRISTINE MASSEY, Institute for Research in Cognitive Science, University of Pennsylvania

• RICHARD E. MAYER, Department of Psychology, University of California, Santa Barbara

• KENT McGUIRE, Southern Education Foundation, Atlanta, Georgia

• P. DAVID PEARSON, Graduate School of Education, University of California, Berkeley

• EDWARD A. SILVER, School of Education and Mathematics Department, University of Michigan

Committee Charge

• Define the set of key skills referred to as “deeper learning,” “21st century skills,” and by other labels

• Describe how the skills relate to each other and to the learning of reading, mathematics, and science and engineering

• Review research on their importance for positive adult outcomes

• Discuss how to teach and assess them

• Identify features of interventions that develop them

Three Domains of Competence

• Cognitive: reasoning and memory

• Intrapersonal: self-management

• Interpersonal: expressing ideas and interpreting and responding to others’ messages

• The 3 domains are intertwined

8

9

10

11

Constructs vs. measures

12

Persistence (Conscientiousness)

Constructs vs. measures

Persistence (Conscientiousness)

SELF RATINGS “I work on a task until I am finished—”rarely” “sometimes” often” “always or almost always”

TEACHER RATINGS “X completes his/her assignments” “true” “not true””

ADMINISTRATIVE RECORDS Missed assignments___; Tardiness ___

SITUATIONAL JUDGMENT “You have a test the next day and don’t feel fully prepared. You are very tired and you are not thinking clearly. What do you do?___

BEHAVIORAL INTERVIEW “Tell me about a time when you had to persist on a task despite many barriers in your way”

PERFORMANCE TESTS Length of time spent adding columns of numbers before requesting a break

Some classics on the construct vs. measure distinction • “Operationism and the concept of perception”

• Garner, Hake, & Eriksen, 1956

• “Convergent and discriminant validation by the multitrait-multimethod matrix”

• Campbell & Fisk, 1959

• Type-token distinction (logic, linguistics, computer programming), concept vs. instances of the concept

• Psychometrics: Factor analysis; Reliability, true score, generalizability theory; MIMIC models; latent vs. observed variable; structural vs. measurement model; structural equation modeling

• Spearman, 1910; Thurstone, 1935; Cronbach, Rajaratnman, & Gleser, 1963; Hauser & Goldberger, 1971; Joreskog & Goldberger, 1975; Joreskog & Sorbom, 1981;

14

15

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

16

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Almost all interpersonal and intrapersonal measurement is based on rating scales (Likert, 1932)—they’re very useful

• But these have problems (reference-group effects, social desirability, response style effects)

• Anchoring Vignettes (“AVs”) seem to address some of these problems (Kyllonen & Bertling, 2014)

• Statistical corrections may also help (e.g., Khorramdel, et al, in press)

• Likert scale measurement is so bounded to “personality” that they’re often confused! (e.g., “personality change” literature)

17

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Others’ ratings are also widely used (Behavioral Anchored Ratings Scales are a structured version), and are more predictive of outcomes than self ratings are

• Poropat, 2014; Oh, Wang, & Mount, 2011; Connelly & Ones, 2010)

• ETS PPI (Kyllonen, 2008; Klieger, 2015) was used for grad school admissions, but is being discontinued

18

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Forced-choice methods are more predictive of educational and workforce outcomes than rating scales methods (Salgado & Tauriz, 2012)

• Recent work we have done in a graduate education program has confirmed this (Naemi et al., 2014)

• New, IRT-based scoring models provide more reliable, and valid measurement (Stark et al., 2005; Brown and Maydeu-Olivares, 2013)

19

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Letters of recommendation are widely used in higher education and have predictive validity beyond other sources (Kuncel, Kochevar, & Ones, 2014)

• Faculty members use letters to infer many qualities about applicants, albeit subjectively (Kyllonen, 2008)

20

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Situational judgment tests have been used in higher education (Lievens & Sackett, 2012; AAMC planning; Oswald et al., 2004)

21

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Interview use is increasing due to technology and remote interviewing

• Some meta-analyses document the features of interviews that predict outcomes (Kell et al., in preparation)

22

L

iker

t S

cale

(in

cl.

AV

s)

F

orc

ed C

ho

ice

B

iod

ata

P

erso

nal

Sta

tem

ent

L

iker

t S

cale

(in

cl.

BA

RS

)

F

orc

ed C

ho

ice

L

ette

r o

f R

ecom

men

dat

ion

T

ext

pro

mp

t-R

ank

Res

ponse

V

ideo

/An

imat

ed P

rom

pts

In

terv

iew

Res

po

nse

F

ace

to f

ace

-- B

ehav

iora

l

B

ehav

iora

l vid

eo i

nte

rvie

w

C

oll

abo

rati

on

/Ro

le P

lay

M

ult

iple

Ch

oic

e

S

hort

An

swer

, E

ssay

In

tera

ctiv

e (g

ame)

Problem Solving, Critical Think. ─ ─ ─ ─ ─ ─ l l l l

Idea Production, Creativity l ─ l ─ l l ◦ l l

Information Technology l ─ ─ ─ l ─ l ◦ ◦ ◦ ◦ ◦ l l

Scientific Inquiry Skills l ─ ─ ─ l ─ l l ◦ ◦ ◦ ◦ ◦ l l

Global/Cultural Skills l ─ l ─ l l ◦ ◦ ◦ ◦

Communication Skills-Oral l ◦ ◦ l ─ l l l l l l

Communication Skills-Written l l ─ l ─ ─ ─ l l

Leadership Skills l l ─ l l l l ◦ ◦

Teamwork, Collaboration Skills l l ─ l l l l

Self-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦

Emotional-Regulation Skills l l ◦ ◦ l ─ l l ◦ ◦ ◦ ◦ ◦ ◦

Interests & Attitudes l l l ─ l ◦ ◦ ◦ l ◦ ◦ ◦ ◦

Career Aspirations l l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Subjective Well Being l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Health l ◦ ◦ l ─ l ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

─

l

◦

Tests

No research in this area

Inte

r-

per

son

al

Intr

a-

per

son

al

Assessments could be developed but there is no need for them (better assessments are available)

Assessments are available and widely used (mature technology)

Assessments are being developed or in experimental usage (developing technology)

Assessments are in early exploration stages (research status)

Co

gn

itiv

eSelf Ratings

Others'

Ratings

Situational

Judgment Interviews

• Lots of interest in “tests” or “performance measures” of inter and intrapersonal skills

• “Idea Production” Creativity (Frederickson & Ward, 1975)

• “Grit game” (Alan, Boneva, Ertac, 2015; Gerhards & Graverts, 2015)

• Emotional Intelligence (Mayer et al., 2005)

• Collaborative problem solving (Wooley et al., ; PISA 2015; von Davier et al., 2016)

Another category missing from the taxonomy: Administrative Records

• General noncognitive index

• Evaluated OneGoal (designed to improve college enrollment and persistence by improving interpersonal and intrapersonal skills) (Kautz & Zanoni, 2014).

• Showed that a noncognitive index (grades, credits earned, absences, and disciplinary infractions) accounted for some of the OneGoal effectiveness

• Also Jackson (2013) & Kautz & Heckman (in prep, Argentina)

• Procrastination • Enrollment latency (Novarese and Di Giovinazzo 2013)

• Homework delays (Novarese and Di Giovinazzo 2015)

• Web-based homework (Schiming, 2012)

23

Summary

• “Education for Work and Life” lays out the constructs provides a good rationale from several perspectives

• Measurement and constructs are separate issues

• Measurement continues to be the challenge and the barrier to implementation

• Likert scale measurement (mostly self, but also other) is almost synonymous with interpersonal and intrapersonal constructs. This is a problem.

• Meta-analyses are beginning to tell us important lessons about measurement (e.g., forced choice> ratings; others> self)

• There is lots we do not yet know, and developments are happening rapidly, in K-12, higher education, and workplace

24

References

• Alan, S., Boneva, T., & Ertac, S. (2015). Ever Failed, Try Again, Succeed Better: Results from a Randomized Educational Intervention on Grit (No. 2015-009).

• Bennett, R. E., & Rock, D. A. (2005). Generalizability, validity, and examinee perceptions of a computer-delivered formulating hypotheses test. Journal of Educational Measurement, 32(1), 19-36.

• Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460-502.

• Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological bulletin, 56(2), 81.

• Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press.

• Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092-1122.

• Cronbach, L.J. (1970). Essentials of psychological testing. New York: Harper & Row.

• Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†. British Journal of Statistical Psychology, 16(2), 137-163.

• Davies, M., Stankov, L., & Roberts, R. D. (1998). Emotional intelligence: In search of an elusive construct. Journal of Personality & Social Psychology, 75, 989-1015.

• Duckworth, A.L., & Gross, J.J. (2014). Self-control and grit: Related but separable determinants of success. Current Directions in Psychological Science, 23, 319-325. doi: 10.1177/0963721414541462

• Educational Testing Service (2014). Relationship between Big 5 and Academic and Workforce Outcomes. Retrieved from https://www.ets.org/s/workforce_readiness/pdf/21334_big_5.pdf 2/16/2015.

• Frederiksen, N., & Ward, W. C. (1975). MEASURES FOR THE STUDY OF CREATIVITY IN SCIENTIFIC PROBLEM‐SOLVING. ETS Research Bulletin Series, 1975(1), 1-4.

• Garner, W. R., Hake, H. W., & Eriksen, C. W. (1956). Operationism and the concept of perception. Psychological review, 63(3), 149.

• Gerhards, L., & Gravert, C. (2015). Grit Trumps Talent? An experimental approach.

• Goldberger, A. S., & Hauser, R. (1971). The treatment of unobservable variables in path analysis. Sociological methodology, 3(8), 1-8.

• Heckman, J. J., Humphries, J. E., & Kautz, T. (Eds.). (2014). The myth of achievement tests: The GED and the role of character in American life. University of Chicago Press.

• Heckman, J. J. & Kautz, T. D. (2014). Hard evidence on soft skills. Working Paper 18121. Retreived from http://www.nber.org/papers/w18121.pdf, February 19, 2015. National Bureau of Economic Research, Cambridge MA.

25

References • Hoyle, R.H. (Ed.) (2010). Handbook of Personality and Self-Regulation. West Sussex: Wiley-Blackwell.

• Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70(351a), 631-639.

• Jöreskog, K. G., & Sörbom, D. (1981). LISREL V: Analysis of linear structural relationships by maximum likelihood and least squares methods. University of Uppsala, Department of Statistics.

• Kautz, T., & Zanoni, W. (2014). Measuring and fostering non-cognitive skills in adolescence: Evidence from Chicago Public Schools and the OneGoal Program. Unpublished manuscript, Department of Economics, University of Chicago, Chicago, IL.

• Khorramdel, L., Von Davier, M.,Bertling, J., Roberts, R., & Kyllonen, P.C. (in press). Applying a New IRT approach to Test & Correct for Response Styles in PISA Background Questionnaire Data: A Feasibility Study.

• Klieger, D.M, Ezzo, C., Bochenek, J., Cline, F.A., & Li, C. (2015). The Predictive Validity of Non- Cognitive Skills for Graduate and Professional Student Success: Some Initial Findings. Presented to Educational Testing Service, Princeton NJ.

• Krathwohl, D. R.; Bloom, B. S.; Masia, B. B. (1964). Taxonomy of educational objectives: The classification of educational goals. Handbook II: the affective domain. New York: David McKay Company.

• Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A Meta‐analysis of Letters of Recommendation in College and Graduate Admissions: Reasons for hope. International Journal of Selection and Assessment, 22(1), 101-107.

• Kyllonen, P. C. (2008). The research behind the ETS personal potential index (PPI). Princeton, NJ: ETS

• Kyllonen, P.C. (2016). Socio-emotional and self-management variables in learning and assessment. In A. Rupp, J. Leighton (Eds.), Handbook of Cognition and Assessment, New York: Wiley/Blackwell.

• Kyllonen, P. C., & Bertling, J. P. (2013). Innovative questionnaire assessment methods to increase cross-country comparability. In L. Rutkowski, M. von Davier, & D. Rutkowski, (eds.) Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis. Boca Raton: CRC Press.

• Likert, R. (1932). A technique for the measurement of attitudes. Archives of psychology.

• Lindqvist, E. and Vestman, R. (2011) The labor market returns to cognitive and non-cognitive ability: Evidence from the Swedish enlistment. American Economic Journal: Applied Economics 3, pp. 101-128

• Liu, O., Bridgeman, B. & Adler, R. (2013). Learning outcomes assessment in higher education: Motivation matters. Educational Researcher, 41, 352-362.

• MacCann, C., Joseph, D., Newman, D., Roberts, R. (2014). Emotional Intelligence Is a Second-Stratum Factor of Intelligence: Evidence From Hierarchical and Bifactor Models. Emotion, 14(2), 358-374.

• Mayer, J. D., Caruso, D. R., & Salovey, P. (2005). The Mayer-Salovey-Caruso Emotional Intelligence Test–Youth Version (MSCEIT-Youth Version) Research Version. Toronto, Ontario, Canada: Multi-Health Systems.

• McAbee, S. T., & Oswald, F. L. (2013). The criterion-related validity of personality measures for predicting GPA: a meta-analytic validity competition. Psychological Assessment, 25 (2), 532-544.

26

References • Mumford, M. D., Marks, M. A., Connelly, M. S., Zaccaro, S. J., & Johnson, J. F. (1998). Domain- based scoring in divergent-thinking tests:

Validation evidence in an occupational sample. Creativity Research Journal, 11(2), 151-163.

• Noftle, E. E., & Robins, R. W. (2007). Personality predictors of academic outcomes: big five correlates of GPA and SAT scores. Journal of Personality and Social Psychology, 93(1), 116-130.

• Novarese, M., & Di Giovinazzo, V. (2013). Promptness and academic performance.

• Novarese, M., & Di Giovinazzo, V. (2015). Not Through Fear But Through Habit. Procrastination, cognitive capabilities and self-confidence (No. 179). Institute of Public Policy and Public Choice-POLIS.

• Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance: A quantitative summary of over 60 years of research. Perspectives on Psychological Science, 7(4), 384-403.

• O’Connor, M. C. & Paunonen, S. V. (2007). Big Five personality predictors of post-secondary academic performance. Personality and Individual Differences, 43, 971–990.

• OECD (2013). OECD guidelines on the measurement of subjective well-being. Retrieved from http://www.oecd.org/statistics/Guidelines%20on%20Measuring%20Subjective%20Well-being.pdf, February 15, 2015. Paris: Office of Economic Cooperation and Development.

• Oh, I. S., Wang, G., & Mount, M. K. (2011). Validity of observer ratings of the five-factor model of personality traits: A meta-analysis. Journal of Applied Psychology, 96(4), 762-773.

• Pellegrino, J. W., & Hilton, M. L. (Eds.). (2013). Education for life and work: Developing transferable knowledge and skills in the 21st century. National Academies Press.

• Poropat A. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, Vol. 135(2), 322-338.

• Poropat, A. (2014). Other-rated personality and academic performance: Evidence and implication. Learning and Individual Differences.

• Richardson, M., Abraham, C., Bond, R. (2012). Psychological correlates of university students' academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138, 353-387.

• Richardson, M., Abraham, C., Bond, R. (2012). Psychological correlates of university students' academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138, 353-387.

• Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 271–288.

• Salovey, Peter; Mayer, John; Caruso, David (2004), Emotional Intelligence: Theory, Findings, and Implications, Psychological Inquiry: 197–215

• Saucier, G. (2000). Isms and the structure of social attitudes. Journal of Personality and Social Psychology, 78, 366−385.

• Saucier,G., & Goldberg,L.R. (2001). Lexical studies of indigenous personality factors: Premises, products, and prospects. Journal of Personality, 69(6), 847-880.

27

References

• Schiming, R. C. (2012). Patterns of homework initiation for web-based activities in economics: A study of academic procrastination. Journal for Economic Educators, 12(1), 13-25.

• Schmitt, N., Oswald, F. L., Kim, B. H., Gillespie, M. A., & Ramsay, L. J. (2004). The Impact of Justice and Self‐Serving Bias Explanations of the Perceived Fairness of Different Types of Selection Tests. International Journal of Selection and Assessment, 12(1‐2), 160-171.

• Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184-203.

• Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 1904-1920, 3(3), 271-295.

• Thurstone, L. L. (1935). The vectors of mind.

• Von Davier, A., Zhu, M., & Kyllonen, P. (in press). Innovative Assessment of Collaboration. Springer.

• Wooley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330, 686-688.

28