Doug Reynolds, Development Dimensions International

Presenter:

Big Data and Talent Management:

Using assessment and technology to run better organizations

Presented by:

Doug Reynolds, Ph.D. Senior Vice President & CTO Development Dimensions International (DDI)

Big Data:

Old Wine or New Opportunity?

Presented by:

Doug Reynolds, Ph.D. Senior Vice President & CTO Development Dimensions International (DDI)

“Big Data” About People is Hot!

Anatomy of a trend

Reference to

Moneyball

Uninformed

over-

generalization

Pivot point

Anatomy of a trend

Review of

common

problems

Anatomy of a trend

Discovery of

known

knowledge

Nuggets of

insight from

new

technologies

What does big data mean for I-O and HR?

• A visit to the dust bowl

• A sea of spurious correlation

• High school science

• New insights from new tools

A few definitions

• Big data: Large quantity and variety of data

generated through internet-based systems; often

involves data from multiple systems (3Vs).

• Digital exhaust: trace information remaining

after use of an online tool, often irrelevant to the

purpose of the tool.

• HR analytics: organizationally relevant statistics

regarding people associated with the

organization

Roadmap for the discussion

• Why are we talking about this now?

• The promise of big data in HR

• A few examples

• Old wine? New opportunity?

A few important trends

Labor Market Management Technology

• Talent is a differentiator

• Aging workforce

• Skill gaps

• Automation

• Globalization

• Outsourcing

• Interoperability

• Services architecture

• Cloud computing

A (simplified) view

of the evolution

of business software

Start with an inefficient business process…

Input

Step 1

Step 2

Step 3

Step 4

Output

… design software to improve it

Input

Step 1

Step 2

Step 3

Step 4

Output

Simplify, automate,

increase availability,

globalize, etc.

It doesn’t take much to get started…

Competitors may capture different parts of the process

Company 1

Company 2

The next challenge: add value beyond your step

Interconnect with other

software tools to automate

more of the business

process

Acquire, merge, or build to own the whole process

Attempt to support the

whole process:

• Build more pieces

• Buy your neighbor

• Sell out

• Or, go out of

business

Once you own one process, you get hungry for others…

Process: A B C D

Once you own one process, you get hungry for others…

Process: A B C D

Interconnections allow for more insight and strategic value

Within

Process

A B C D

Across

Processes

In the HR context:

Within

Process

Across

Processes

Big data about people

Recruit

Hire

Train Manage

Promote

The promise of big data for talent

management

Strategic Impact

Process Automation

Insight

Predictive Hiring Analytics

Manager

Satisfaction

with Quality

of Candidates Ratio Offers to

Acceptance and

Diversity of

New Hires Candidate

feedback

on the hiring

process

Confidence

of Hiring Manager

in the New Hire

and Confidence

of the New Hire

that they are in

the Right Job

Job Performance

and

Engagement

Source Final

Interview Job Offer First Day

on the Job

6 Months

on the Job

1 year

on the job Assessment

Candidate

Source

Assessment

Data for

Individual

and Group

A pervasive issue: assessment rigor

Talent quality?

1 2 3 4 5

Examples of Assessment-driven Analytics

Example 1: selection testing

• Selection test for graduate hiring

• Relevant and effective across cultures

• Strong security (difficult to cheat on)

• Strong predictor of performance

• Available anytime, anywhere

• Brief

Test features

• Figural reasoning: measure of reasoning ability,

critical thinking, and problem-solving

• Non-verbal/graphical items

– No translations

– Applicable regardless of candidate reading level

– Culture-free/fair for all candidate groups

– Allows for comparisons across cultures/countries

Internet-based computer adaptive testing (CAT)

CAT addresses several common issues:

Test Security Items drawn from extensive bank;

low item exposure rates

Cheating

Different combination of items for

each candidate; no single key

available to be used for cheating

Length of

Candidate

Experience

Compared to traditional tests,

shorter test time but superior

precision

CAT: Development Process

Calibration Research

• 200,000+ candidates for entry-

level professional jobs, globally

• Items researched via internet

delivered test forms

• Test timing and question

functioning developed from

response data

• Ensured the test is inclusive to all

candidates globally Then, criterion-related validation study conducted

Results from CAT validation

Criterion Validity Coefficient

Composite Performance 0.36 (0.29)**

Gathering Information 0.18 (0.14)**

Reviewing and Analyzing Information 0.34 (0.27)**

Decision Making 0.33 (0.26)**

Strategic and Operational Agility 0.20 (0.16)**

Innovation 0.20 (0.16)**

Potential 0.27 (0.21)**

Adaptability 0.25 (0.20)**

Note. N=596 ** p < 0.01. Validity coefficients have been corrected only for unreliability in the criterion using a reliability estimate of 0.63. Values in parentheses represent uncorrected validity coefficients.

Interview scores by CAT score group

0

500

1000

1500

2000

2500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

RT scores with passing RS score RT scores with or without RS scoreNext step scores with passing CAT Next Step scores with or without passing CAT

Assessment-driven metrics

• Employer brand index

• Recruiting effectiveness metrics

Test scores + HRIS data: Turnover survival analysis

Key characteristics:

• Day-in-the-life format

• Technology driven

• Live interactions

• Deployed globally

Example 2: Tech-facilitated assessment center

72

87

25

100

98

88

95

Adjustment

Ambition

Sociability

Interpersonal Sensitivity

Prudence

Inquisitiveness

Learning Orientation

Leadership Effectiveness Inventory

33

10

55

24

15

13

11

53

69

24

72

Volatile

Argumentative

Risk Averse

Imperceptive

Avoidant

Arrogant

Impulsive

Attention Seeking

Eccentric

Perfectionistic

Approval Dependent

Leadership Challenge Inventory

Sample Output

PE

RS

ON

AL

ITY

PA

TT

ER

NS

CO

MP

ET

EN

CIE

S

Karen Gates Vice President , Operations

Started: 02.04.2000

Previous Position: Director, Operations

Education: MBA, Wharton Business School

Known Aspirations: VP of Eastern Region

Interpersonal Skills

Compelling Communication

Cultivating Networks

Navigating Politics

Influence

P

S

D

P

Business Management

D

S

P

P

P

P

Building Organizational Talent

Driving Execution

Financial Acumen

Operational Decision Making

Entrepreneurship

Establishing Strategic Direction

Leadership

S

P

S

P

Personal Competencies

Leading Change

Coaching and Developing Others

Selling the Vision

Empowerment/Delegation

P

S

Executive Disposition

Passion for Results

Multi-organization analyses

16,000+ Executives

Strengths

Driving for Results

Executive Disposition

Communicating with Impact

Decision Making

Customer Focus

Strengths

Establishing Strategic Direction

Financial Acumen

Entrepreneurship

Building Talent

Strong Managers

Strategic Leaders

Vary by:

Level?

Industry?

Experiences?

Personality profile?

Company Performance

Prior Assessments

HRIS

Company Profiles

Low

High

Med

Coaching

Customer

Focus

Business

Savvy

Empowerment

2nd

Level

4th

Level

3rd

Level

Level differences in leadership

Example using alternative methods

16,000+ Executives

Strong Managers

Strategic Leaders

Vary by:

Personality profile?

Machine learning:

not your typical PSYCH 650 class

Example: Random Forests

Strategic Profile (Y/N)?

Med/Lo Risk Averse Hi Risk averse

Hi Sociability Lo Sociability

Hi Ambition Lo Ambition

Lo Risk Averse Med Risk averse

Example results:

Random Forests & Logistic Regression

Rank Random Forest Logistic Regression

1 Independent Thinker/

Strong Decision Maker

Strong Interpersonal

Relations

2 Energetic: Drives Self and Others Independent Thinker/

Strong Decision Maker

3 Strong Interpersonal Relations Conflict Averse (-)

4 Strategic/Creative B/W Thinking Style

5 Self-Promoting Emotionally Detached

6 Emotionally Unpredictable Thoughtful/Planful

= Similar Rank

= Close (Top 10 in opposing model)

= Dissimilar Rank

Error rate: 12.5% Error rate: 19%

Big Data: Old wine or new opportunity

The Answer:

Big data:

Old wine

and

New opportunity

(with some significant challenges ahead)

“Big Data” About People

Challenges in practice:

• Software is often an expensive hollow shell

• People data can be of poor quality

• New data sources are not well understood

• Interpretations can be terribly flawed

• Managers defer to gut instinct

Big Data: Big Skills Required

• Complex data analysis and modeling

• Knowledge of people & systems

in organizations

• Theory building and testing

• Communication and

action planning

“Big Data” About People

Opportunities in practice:

• Interconnections across steps add new

insights and strategic value

• Strong theory, modeling, hypothesis testing

are essential to extract meaning and order

from huge complexity

• Insight about people can be packaged to

better inform organizational strategy

Post-SIOP survey:

“What are you most excited about?”

SIOP Taskforce on Big Data

Key areas for action:

• Meaning and definition of big data

• Training in research methods

• Theory generation & testing

• Interdisciplinary linkages

• Applications to validation practices

• Legal and ethical issues

• Education and awareness within SIOP

• Overcoming resistance

Will this trend leave I-O behind?

“In the world of Big Data, companies can assess

people on real world performance. A flurry of new

companies… are now skipping I/O psychology and

helping recruiters source candidates by analyzing

their social data. While a test is a good way to

understand someone, so is looking at everything

they’ve ever posted on the internet.”

Josh Bersin, Forbes, Oct 2013.

Concluding Thoughts:

• New opportunities are emerging for I-O and

the role of assessment

• Potential for better people strategy

if we can overcome common barriers

• We have an obligation to respond to the popular

trend

Thank you!

Thank you.

Big Data:

Old Wine or New Opportunity?

Doug Reynolds February, 2014