Evidence-Based Management:
Valid and Reliable Organizational Data
Denise M. RousseauH.J. Heinz II University Professor of Organizational Behavior
Carnegie Mellon [email protected]
EBMgt Overcomes Limits of Unaided Decisions
Bounded Rationality
The Small Numbers Problem of Individual Experience
Prone to See Patterns Even in Random Data
Critical Thinking & Ethics
Research• Large Ns > individual
experience• Controls reduce bias
Relevant Org’l Facts• Reliable and valid• Interpreted using
appropriate logics Decision Supports
The “Human” ProblemEvidence-Based Practice
1. Get Evidence into the Conversation
2. Use Relevant Scientific Evidence
3. Use Reliable and Valid Business Facts
4. Become “Decision Aware” and Use Appropriate Processes
5. Reflect on Decision’s Ethical and Stakeholder Implications
Five Good EBMgt Habits
Different Ways of Thinking about Problems
4
Problem Prototype
Process of Thought
Solution
Rearrangement of Elements
Local Optimization
Mechanical Thinking Intuitive Thinking Strategic Thinking
Transformation or changed configuration
Analysis of Essence
5
Example (1 of 4): the number of operations centres at a major bank were reduced by 82%
Consolidation in Operating Units over 2 Years
Call Centers
Credit Granting
Credit Management and Recoveries
Business Service Centers
Personal Service Centers
Operations Service Centers
35
15
20
50
15
7
142
4
4
4
5
5
3
2582% reduction
Centres with over 200 FTEsCentres with under 200 FTEs
65%35%
1996 1998-9
Example of Consolidation
6
Example (2 of 4): intuition suggests that the smaller centres should be less efficient than the large centres due to scale economie
Scale Curve for Traditional Business
R 2 = 59%
0%
10%
20%
30%
40%
50%
60%
- 200 400 600 800
FTEs in Center
Expected scale curve
Indicator of efficiency
Indicator of size of centre
Expected Result
Non-staff cost/All cost
7
Example (3 of 4): however, the reverse was found to be true
(1) total cost/FTE vs. FTEs shows same relationship, but with r2 of 44%Source: interviews, financial Q1 1999 operating results; call center economics excludes some centers for which data was not available
Contrary to Expectations, Units Don’t Show Expected Scale Benefits (1)…
R 2 = 59%
0%
10%
20%
30%
40%
50%
60%
- 200 400 600 800FTEs in Center
Actual
Actual Result
Expected scale curve
Non-staff cost/All cost
8
Example (4 of 4): this radically altered the course of the project
Assuming that Scale Economies Existed
· Consolidate further - further worsening the situation
· Aim for generalized headcount cuts to improve financials - further stretching staff
Knowing that Reverse Scale Economies Existed
· Understand why reverse scale economies exist (reason: inconsistent policies, practices and processes layered on top of each other from consolidated centres)
· Set in place programs to fix the cause of the problem (solution: standardize policies, practices and processes and reduce headcount and cost accordingly)
Successful Engagement Possible Disaster
Impacts of the Analysis
9
85
An interesting example of the need for facts, correct analysis and presentation
Dots indicate O-ring damage for 24 successful space shuttle launches prior to the Challenger failure. Challenger was launched on January 28, 1986 with 31 degree forecast temperature.
Nu
mb
er
of
da
ma
ge
d O
-rin
gs
p
er
lau
nc
h 1
2
3
0
Temperature - Degrees Fahrenheit
30 35 40 45 50 55 60 65 70 75 80
Sources: 1) From Edward Tufte, Visual Explanations, MS 1991.2) Report of the Presedential Commission on the Space Shuttle Challenger Challenger Accident (Washington DC, 1986) volume V, pages 895-896, ref. 2/26 1 of 3 and ref. 2/26 2 of 3.3) Siddhartha R. Dalal, Edward B. Fowlkes and Bruce Hoadley, “ Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure.” Journal of the American Statistical Association, December 1989, page 946.
Best “Available” Business Facts Should Present Representative Numbers Sampled From
Entire Population
Be Interpretable In Context of Use
Provide Key Indicators for Business Decisions
Inform Well-thought Out Logics
Make Sense to Relevant Decision Makers
Use Reliable and Valid Business Facts
Best “Available” Business Facts Present Representative Numbers Sampled From
Entire Population (No Cherry Picking!)
NOT biased single or isolated cases (Look at TOTAL successes & failures not just one or the other)
NOT based on small sample sizes (Aggregate or combine small units to increase reliability and reduce random variation)
Use Reliable and Valid Business Facts
# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient safety worse in Unit 1?
Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10% turnover. Is retention better in one?
McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?
How to Interpret Business Facts
Organizational Facts Take Many Forms
Data Information Knowledge Problem Solutions
DATA
Raw Observations “Your score is six”“Eight medication errors occurred”
Ignores total possible#items on test
Neglects Base RateHow many total administrations of medication?
#errors plus #non-errorsOrganizational Context
How many in same department? By Same Person?
DATA
Challenges: Are data reliable? Complete?
Unbiased?
INFORMATION
Making Data Interpretable Possible Totals or Percentages
“6 out of 10” “60% correct”Base Rate Represented
“8 errors out of 450 administrations”
Organizational Context ConnectedWhen? Where? By Whom?
4 errors made by same person are likely due to different factors than are 4 errors made by different individuals in different departments
Using Information Effectively
Informs Well-thought Out Logics that Decision Makers are Skilled in Using
Based on critical thinking, consideration of alternatives, and systematic evidence regarding appropriateness
InputsProcessesLead OutcomesLagged Results
KNOWLEDGE
KNOWLEDGEKNOWLEDGE PROVIDES KEY INDICATORS FOR ORGANIZATIONAL DECISIONS
DIAGNOSISIs 18% turnover a problem? A good thing?
KNOWN IMPACT ON KEY OUTCOMESWhat’s the success rate of applicants scoring at or
above 60% on a test in the first year on the job?
Best “Available” Business Facts Should INFORM ON THE WELL-BEING, HEALTH AND
PERFORMANCE OF THE ORGANIZATION
Lead and lagged indicators reflecting organization’s performance pathways– Lead indicators: critical conditions to be managed in order to
achieve important (lagged) outcomes (customer satisfaction)
– Lagged indicators: important business outcomes (sales growth)
Use Reliable and Valid Business Facts
Best “Available” Business Facts Should Inform relevant decision makers
Using frames, definitions, and logics they understand
Focus of decision maker attention will largely be influenced by available facts (data, metrics)
– Both unit-specific data and organization-wide
– If too narrow, other goals may be neglected
– If too broad, may be forced to simplify
– If not measured, it cannot be rewarded, if not reward, it won’t occur.
Use Reliable and Valid Business Facts
Use Reliable and Valid Business FactsFocus of attention is influenced by metrics
Division A’s
Metrics
Budget Compliance
1st,2nd,3rd & 4th Qtr
Profitability
1st,2nd,3rd & 4th Qtr
Division B’s Annual Metrics
Customer SatisfactionEmployee RetentionStaff DevelopmentReturn on AssetsPortion of Sales from
Recently Developed Products
The QUESTION you are trying to answer determines the ANALYSIS:
How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?
22
• A count or percentage
• Always an estimate, not “truth”
• Report confidence interval (the likely range the true number falls within)
• A count or percentage
• Always an estimate, not “truth”
• Report confidence interval (the likely range the true number falls within)
Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
The QUESTION you are trying to answer determines the ANALYSIS:
How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?
23Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Show if factor really led to outcome change, using:
Comparison of averages
Time series -- % change over time
Before & After (pre/post) tests
Compare to groups without factor – difference scores (t-tests)
• Show if factor really led to outcome change, using:
Comparison of averages
Time series -- % change over time
Before & After (pre/post) tests
Compare to groups without factor – difference scores (t-tests)
The QUESTION you are trying to answer determines the ANALYSIS:
How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?
24Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Regression analysis tells which of a set of factors are significantly related
• Suitable for two kinds of data:
Dichotomous (College YES 1 NO 0)
Continuous (years of education)
• Regression analysis tells which of a set of factors are significantly related
• Suitable for two kinds of data:
Dichotomous (College YES 1 NO 0)
Continuous (years of education)
The QUESTION you are trying to answer determines the ANALYSIS:
How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?
25Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Compare with past number
Example: 2012 values divided by 2011’s
• What controls do we need to really know what is what?
• Compare with past number
Example: 2012 values divided by 2011’s
• What controls do we need to really know what is what?
The QUESTION you are trying to answer determines the ANALYSIS:
How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?
26Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• When and how would you act on it?• When and how would you act on it?
Turning evidence into practice
Evidence-Based Management:Closing the gap between research and practice
Turning Evidence into Practice & Practice into Evidence
Got Evidence? References
M. Blastland & A. Dilnot (2007) The Tiger that Wasn’t: Seeing through a World of Numbers. London, Profile Books.
D. Kahneman (2011) Thinking, Fast and Slow. New York: Farrar, Straus & Giroux.
D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.
D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource Management Journal, 21, 221-235.
N.Silver (2012) The Signal and the Noise: Why So Many Predictions Fail but Some Don’t. New York: Penguin.
Illustration--Discuss with your seatmates…
What indicators does your organization most commonly use to make important decisions?
Are these the “best business facts” you need to make these decisions?
What indicators would be more useful, if you could get them??
Use Reliable and Valid Business Facts