Evidence-Based Management: Valid and Reliable Organizational Data Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior Carnegie

Evidence-Based Management:

Valid and Reliable Organizational Data

Denise M. RousseauH.J. Heinz II University Professor of Organizational Behavior

Carnegie Mellon [email protected]

EBMgt Overcomes Limits of Unaided Decisions

Bounded Rationality

The Small Numbers Problem of Individual Experience

Prone to See Patterns Even in Random Data

Critical Thinking & Ethics

Research• Large Ns > individual

experience• Controls reduce bias

Relevant Org’l Facts• Reliable and valid• Interpreted using

appropriate logics Decision Supports

The “Human” ProblemEvidence-Based Practice

1. Get Evidence into the Conversation

2. Use Relevant Scientific Evidence

3. Use Reliable and Valid Business Facts

4. Become “Decision Aware” and Use Appropriate Processes

5. Reflect on Decision’s Ethical and Stakeholder Implications

Five Good EBMgt Habits

Different Ways of Thinking about Problems

4

Problem Prototype

Process of Thought

Solution

Rearrangement of Elements

Local Optimization

Mechanical Thinking Intuitive Thinking Strategic Thinking

Transformation or changed configuration

Analysis of Essence

5

Example (1 of 4): the number of operations centres at a major bank were reduced by 82%

Consolidation in Operating Units over 2 Years

Call Centers

Credit Granting

Credit Management and Recoveries

Business Service Centers

Personal Service Centers

Operations Service Centers

35

15

20

50

15

7

142

4

4

4

5

5

3

2582% reduction

Centres with over 200 FTEsCentres with under 200 FTEs

65%35%

1996 1998-9

Example of Consolidation

6

Example (2 of 4): intuition suggests that the smaller centres should be less efficient than the large centres due to scale economie

Scale Curve for Traditional Business

R 2 = 59%

0%

10%

20%

30%

40%

50%

60%

- 200 400 600 800

FTEs in Center

Expected scale curve

Indicator of efficiency

Indicator of size of centre

Expected Result

Non-staff cost/All cost

7

Example (3 of 4): however, the reverse was found to be true

(1) total cost/FTE vs. FTEs shows same relationship, but with r2 of 44%Source: interviews, financial Q1 1999 operating results; call center economics excludes some centers for which data was not available

Contrary to Expectations, Units Don’t Show Expected Scale Benefits (1)…

R 2 = 59%

0%

10%

20%

30%

40%

50%

60%

- 200 400 600 800FTEs in Center

Actual

Actual Result

Expected scale curve

Non-staff cost/All cost

8

Example (4 of 4): this radically altered the course of the project

Assuming that Scale Economies Existed

· Consolidate further - further worsening the situation

· Aim for generalized headcount cuts to improve financials - further stretching staff

Knowing that Reverse Scale Economies Existed

· Understand why reverse scale economies exist (reason: inconsistent policies, practices and processes layered on top of each other from consolidated centres)

· Set in place programs to fix the cause of the problem (solution: standardize policies, practices and processes and reduce headcount and cost accordingly)

Successful Engagement Possible Disaster

Impacts of the Analysis

9

85

An interesting example of the need for facts, correct analysis and presentation

Dots indicate O-ring damage for 24 successful space shuttle launches prior to the Challenger failure. Challenger was launched on January 28, 1986 with 31 degree forecast temperature.

Nu

mb

er

of

da

ma

ge

d O

-rin

gs

p

er

lau

nc

h 1

2

3

0

Temperature - Degrees Fahrenheit

30 35 40 45 50 55 60 65 70 75 80

Sources: 1) From Edward Tufte, Visual Explanations, MS 1991.2) Report of the Presedential Commission on the Space Shuttle Challenger Challenger Accident (Washington DC, 1986) volume V, pages 895-896, ref. 2/26 1 of 3 and ref. 2/26 2 of 3.3) Siddhartha R. Dalal, Edward B. Fowlkes and Bruce Hoadley, “ Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure.” Journal of the American Statistical Association, December 1989, page 946.

Best “Available” Business Facts Should Present Representative Numbers Sampled From

Entire Population

Be Interpretable In Context of Use

Provide Key Indicators for Business Decisions

Inform Well-thought Out Logics

Make Sense to Relevant Decision Makers

Use Reliable and Valid Business Facts

Best “Available” Business Facts Present Representative Numbers Sampled From

Entire Population (No Cherry Picking!)

NOT biased single or isolated cases (Look at TOTAL successes & failures not just one or the other)

NOT based on small sample sizes (Aggregate or combine small units to increase reliability and reduce random variation)


# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient safety worse in Unit 1?

Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10% turnover. Is retention better in one?

McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?

How to Interpret Business Facts

Organizational Facts Take Many Forms

Data Information Knowledge Problem Solutions

DATA

Raw Observations “Your score is six”“Eight medication errors occurred”

Ignores total possible#items on test

Neglects Base RateHow many total administrations of medication?

#errors plus #non-errorsOrganizational Context

How many in same department? By Same Person?

DATA

Challenges: Are data reliable? Complete?

Unbiased?

INFORMATION

Making Data Interpretable Possible Totals or Percentages

“6 out of 10” “60% correct”Base Rate Represented

“8 errors out of 450 administrations”

Organizational Context ConnectedWhen? Where? By Whom?

4 errors made by same person are likely due to different factors than are 4 errors made by different individuals in different departments

Using Information Effectively

Informs Well-thought Out Logics that Decision Makers are Skilled in Using

Based on critical thinking, consideration of alternatives, and systematic evidence regarding appropriateness

InputsProcessesLead OutcomesLagged Results

KNOWLEDGE

KNOWLEDGEKNOWLEDGE PROVIDES KEY INDICATORS FOR ORGANIZATIONAL DECISIONS

DIAGNOSISIs 18% turnover a problem? A good thing?

KNOWN IMPACT ON KEY OUTCOMESWhat’s the success rate of applicants scoring at or

above 60% on a test in the first year on the job?

Best “Available” Business Facts Should INFORM ON THE WELL-BEING, HEALTH AND

PERFORMANCE OF THE ORGANIZATION

Lead and lagged indicators reflecting organization’s performance pathways– Lead indicators: critical conditions to be managed in order to

achieve important (lagged) outcomes (customer satisfaction)

– Lagged indicators: important business outcomes (sales growth)


Best “Available” Business Facts Should Inform relevant decision makers

Using frames, definitions, and logics they understand

Focus of decision maker attention will largely be influenced by available facts (data, metrics)

– Both unit-specific data and organization-wide

– If too narrow, other goals may be neglected

– If too broad, may be forced to simplify

– If not measured, it cannot be rewarded, if not reward, it won’t occur.


Use Reliable and Valid Business FactsFocus of attention is influenced by metrics

Division A’s

Metrics

Budget Compliance

1st,2nd,3rd & 4th Qtr

Profitability

1st,2nd,3rd & 4th Qtr

Division B’s Annual Metrics

Customer SatisfactionEmployee RetentionStaff DevelopmentReturn on AssetsPortion of Sales from

Recently Developed Products

The QUESTION you are trying to answer determines the ANALYSIS:

How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?

22

• A count or percentage

• Always an estimate, not “truth”

• Report confidence interval (the likely range the true number falls within)

• A count or percentage

• Always an estimate, not “truth”

• Report confidence interval (the likely range the true number falls within)

Evidence-Based Management: Making Evidence-Based Decisions

What’s the Question?



23Evidence-Based Management: Making Evidence-Based Decisions


• Show if factor really led to outcome change, using:

Comparison of averages

Time series -- % change over time

Before & After (pre/post) tests

Compare to groups without factor – difference scores (t-tests)

• Show if factor really led to outcome change, using:

Comparison of averages

Time series -- % change over time

Before & After (pre/post) tests

Compare to groups without factor – difference scores (t-tests)





• Regression analysis tells which of a set of factors are significantly related

• Suitable for two kinds of data:

Dichotomous (College YES 1 NO 0)

Continuous (years of education)

• Regression analysis tells which of a set of factors are significantly related

• Suitable for two kinds of data:

Dichotomous (College YES 1 NO 0)

Continuous (years of education)





• Compare with past number

Example: 2012 values divided by 2011’s

• What controls do we need to really know what is what?

• Compare with past number

Example: 2012 values divided by 2011’s

• What controls do we need to really know what is what?





• When and how would you act on it?• When and how would you act on it?

Turning evidence into practice

Evidence-Based Management:Closing the gap between research and practice

Turning Evidence into Practice & Practice into Evidence

Got Evidence? References

M. Blastland & A. Dilnot (2007) The Tiger that Wasn’t: Seeing through a World of Numbers. London, Profile Books.

D. Kahneman (2011) Thinking, Fast and Slow. New York: Farrar, Straus & Giroux.

D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.

D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource Management Journal, 21, 221-235.

N.Silver (2012) The Signal and the Noise: Why So Many Predictions Fail but Some Don’t. New York: Penguin.

Illustration--Discuss with your seatmates…

What indicators does your organization most commonly use to make important decisions?

Are these the “best business facts” you need to make these decisions?

What indicators would be more useful, if you could get them??


Documents

Evidence-Based Management: Valid and Reliable Organizational Data Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior Carnegie