The evening before the space shuttle Challenger explosion, scientists at NASA caught what they thought was a potentially catastrophic risk with the o-rings considering the unusually cold temperature expected for the morning’s launch. They brought the issue to management attention but failed to influence the final decision enough to stop the launch. Your failure to influence may not cost lives but it could be “catastrophic” for your business. This talk presents my top 10 tips for using data to influence others toward better decisions. Learning outcomes: - The do's and don'ts of visualization - How others lie with data - What makes an effective dashboard - How to communicate uncertainty
- 1. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)PresentingMetricstoGet ResultsPart of the
seminar:Dial M for MetricsTopic presented by:Larry
Maccherone@[email protected]@LMaccherone
[email protected]
2. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)@LMaccherone [email protected] 3.
Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)What?SO What?NOW What?@LMaccherone
[email protected] 4. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)What?@LMaccherone
[email protected] 5. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)@LMaccherone
[email protected] 6. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)SO What?@LMaccherone
[email protected] 7. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)What?@LMaccherone
[email protected] is like photography.Impact is a
function of focus,illumination, and perspective. 8. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)NOW What?@LMaccherone [email protected] 9. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)DontLaunch!@LMaccherone [email protected] 10.
Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Prevent your owndisastrous decisionswith
better visualizationand insight@LMaccherone [email protected]
11. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Criteria forgreat
visualization?@LMaccherone [email protected] 12. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)1. Answers the question, "Compared with
what?@LMaccherone [email protected](SO What?) Trends Benchmarks
13. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)2. Shows causality, or is at least
informed by it.The primary chart used by theNASA scientists showed
O-ringfailure indicators by launch date.Tufte's alternative shows
the samedata by the critical factor,temperature.The fateful shuttle
launch occurredat 36 degree. Tufte's visualizationmakes it obvious
that there is greatrisk for any launch at temperaturesbelow 66
degrees.@LMaccherone [email protected] 14. Presenting Metrics to
Get ResultsDial M for Metrics Seminar (Magennis and Maccherone)3.
Tells a story with whatever it takes.@LMaccherone
[email protected] Still Moving Numbers GraphicsAnd A maybe some
fun 15. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)@LMaccherone [email protected]. Is
credible. Calculations explained Sources Assumptions Who (name
drop?) How Etc. 16. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)Detour toImpact of Agile
Quantifiedto show credibility with IterationLength research(If
there is time)@LMaccherone [email protected] 17. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)5. Is impactful in the social context.Has business
value.@LMaccherone [email protected] 18. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)6. Shows comparisons easily.aka:Save the pie for
dessert@LMaccherone [email protected]: Stephen Few
(Perceptual Edge)
http://www.perceptualedge.com/articles/08-21-07.pdf 19. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)6. Shows comparisons easily. (continued)Can you compare
the market share from one year to the next?Quickly: Which two
companies are growing share the fastest?One pie chart is bad.
Multiple pie charts are worse!!!@LMaccherone [email protected]
20. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)6. Shows comparisons easily.
(continued)@LMaccherone [email protected] about now?Can you
compare themarket share from oneyear to the next? 21. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Example of benchmarks@LMaccherone [email protected]
22. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)@LMaccherone [email protected]. Allows
you tosee the forestANDthe trees. 23. Presenting Metrics to Get
ResultsDial M for Metrics Seminar (Magennis and Maccherone)8.
Informs along multiple dimensions. Is multivariate.@LMaccherone
[email protected] 24. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)@LMaccherone
[email protected]. Leaves in thenumberswherepossible. 25.
Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)10. Leaves out glitter.Examples of how NOT
to do it.@LMaccherone [email protected] 26. Presenting Metrics
to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Top 10 criteria for great visualization1. Answers the
question,"Compared with what?(SO What?)2. Shows causality, or is at
least@LMaccherone [email protected] by it.(NOW What?)3.
Tells a story with whatever ittakes.4. Is credible.5. Is impactful
in the socialcontext. Has business value.6.
Showscomparisonseasily.Credits: Edward Tufte Stephen Few
Gestalt(School of Psychology)7. Allows you to see the forestAND the
trees.8. Informs along multipledimensions. Is multivariate.9.
Leaves in the numbers wherepossible.10. Leaves out glitter. 27.
Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)See
also:http://www.businesscomputingworld.co.uk/top-tips-for-influencing-people-with-data/@LMaccherone
[email protected] 28. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)Big datachanges
everything@LMaccherone [email protected] 29. Presenting Metrics
to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)1. Before big data - Datawarehousing, OLAP and
otherbusiness intelligence tools ...big investment2. After big data
- WarehousesPLUS hadoop and NoSQL ...even bigger investment
andcomplexity@LMaccherone [email protected]. Now. Data
enrichedofferings - Pre-packaged bigdata and machine learning fitto
purpose... dramaticallylower cost and complexityFirms dealing
withanalytics saw everythingchange when big datacame along. Now
anothermajor shift is under way,as the emphasis turns tobuilding
analytical powerinto customer productsand services.~ Thomas H.
Davenport,Harvard Business Review 30. Presenting Metrics to Get
ResultsDial M for Metrics Seminar (Magennis and Maccherone)The
right way to teach calculusTODAY(Larry at whiteboard)@LMaccherone
[email protected] 31. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)The old way of doing
calculusis tomy way of doing calculus- as -The old way of doing
statisticsis totechniques like Bayes and Monte Carlo@LMaccherone
[email protected] 32. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)Bayesian techniques
Use new information to update@LMaccherone [email protected]
knowledge Uses Classifiers Regression Bayesian networks Cautions
Assumes independence (most of the time) You still have to clean the
data You still need a model 33. Presenting Metrics to Get
ResultsDial M for Metrics Seminar (Magennis and Maccherone)What do
you mean, You still need a model.This is the single line in my
Lumenize open sourceanalysis library that is the Bayes
Theorem:p[out] = p * prior.p[o] /(p * prior.p[out] + (1 - p) * (1 -
prior.p[out]))Lumenize.Classifier is over 500 lines long
includingroughly 200 lines dedicated to explaining how this
thisparticular instance models the world (non-parametricmodeling
with v-optimal bucketing, depending upontraining set
size).@LMaccherone [email protected] 34. Presenting Metrics to
Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Benchmarking with Big DataAt Rally: Analyzed data from
10s ofthousands of teams Created the SoftwareDevelopment
PerformanceIndex (SDPI) using principlecomponent analysis,
qualitativecorrelation, etc. Used the SDPI to conductresearch
quantifying theimpact of decisions aroundbehaviors, roles,
motivations,process, etc. (possible additionalpresentation for this
seminar) Used the SDPI to createindustry-wide
benchmark.@LMaccherone [email protected] 35. Presenting Metrics
to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Software Engineering ImprovementRecommendation
Engine@LMaccherone [email protected] 36. Presenting Metrics to
Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Denying the evidence@LMaccherone [email protected]
37. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)We don't see thingsthe way they are.We
seethings the way we@LMaccherone [email protected].~The
Talmud 38. Presenting Metrics to Get ResultsDial M for Metrics
Seminar (Magennis and Maccherone)Next slide is a movie@LMaccherone
[email protected] 39. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)Denying the
Evidence@LMaccherone [email protected] 40. Presenting Metrics to
Get ResultsDial M for Metrics Seminar (Magennis and Maccherone)5
Truths about cognitive bias1. Very few people are immune to it.2.
We all think that we are part of thatTroy and I are of course.
;-)@LMaccherone [email protected] group.3. You can be
trained to get much much better.Douglass Hubbard How to Measure
Anything4. You are always dealing with the present and
yourperception of the past David Snowden5. We do a first fit
pattern match. Not a best fit patternmatch. And we only use about
5% of the informationto do the matching. 41. Presenting Metrics to
Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Calibrated Estimates Decades of studies show that most
managers are statisticallyoverconfident when assessing their own
uncertainty Studies also show that measuring your own
uncertaintyabout a quantity is a general skill that can be taught
with ameasurable improvement Training can calibrate people so that
of all the times theysay they are 90% confident, they will be right
90% of the timeCopyright HDR
[email protected]@LMaccherone [email protected]
42. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)65452175 71 65 581721@LMaccherone
[email protected] ConfidenceStatistical
ErrorUntrained/UncalibratedTrained/Calibrated100%90%80%70%60%50%40%30%682515250%
60% 70%80% 90% 100%Assessed Chance Of Being CorrectPercent
Correct99 # of Responses1997 Calibration Experiment 16 IT Industry
Analysts and 16 CIOs , the analysts were calibrated In January
1997, they were asked To Predict 20 IT Industry events Example:
Steve Jobs will be CEO of Apple again, by Aug 8, 1997 - True or
False?Copyright HDR [email protected] 43.
Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Equivalent Bet calibrationSuppose youre
asked to give a 90% CI for the year in which Newtonpublished the
universal laws of gravitation, and you can win $1,000 in one oftwo
ways:1. You win $1,000 if the true year of publication falls within
your 90% CI.Otherwise, you win nothing.2. You spin a dial divided
into two pie slices, one covering 10% of the dial,and the other
covering 90%. If the dial lands on the small slice, you winnothing.
If it lands on the big slice, you win $1,000.Adjust your 90% CI
until option #1 and option #2 seem equally good to you.Research
suggests that even pretending to bet money in this way willimprove
your calibration.@LMaccherone [email protected] 44. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)@LMaccherone [email protected] tips Don't focus
on consensus.Ritual dissent is a much more successful approach.
Don't reach a conclusion too soon.Someone always sees the disaster
in advance. An FBI agent knew thatsome folks were being trained to
fly but not take off and land. Recognize retrospective
coherence.Everything looks coherent when looking back in time. It
never lookscoherent at the time. Use counter-actuals.What would
have happened, if? What-if? Assign someone the role of devils
advocate.Israels 10th man. 45. Presenting Metrics to Get
ResultsDial M for Metrics Seminar (Magennis and Maccherone)Lying
with statistics@LMaccherone [email protected] 46. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Whats wrong with this?84 86 88 90 92 94@LMaccherone
[email protected]/Jira/GreenhopperLeanKitVendor
Y 47. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Other ways to lie with
statistics@LMaccherone [email protected] Sampling bias Self
selection bias Leading question bias Social desirability bias
Median vs Mean The big zoom-in Correlation does not necessarily
mean causation 48. Presenting Metrics to Get ResultsDial M for
Metrics Seminar (Magennis and Maccherone)Forecasting
101@LMaccherone [email protected] 49. Presenting Metrics to Get
ResultsDial M for Metrics Seminar (Magennis and Maccherone)Next
slide is a movie@LMaccherone [email protected] 50. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)@LMaccherone [email protected] 51. Presenting Metrics
to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Every decision is aforecast!@LMaccherone
[email protected] 52. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)Every decision is a
forecast!You are forecasting that the alternative that you chose is
going to have betteroutcomes than the other alternatives.So The
quality of your decision depends upon two things:1. The QUANTITY
(and thus quality) of alternatives considered Brainstorm to gather
lots of (even crazy) alternatives. Use back-of-napkin models to
rapidly reject lots of bad alternatives. 3x3 = 10. But, be careful
to not to remove whole branches of your decision tree prematurely.
Asingle golden fruit may exist on a branch that is mostly rotten
apples.2. The models used to forecast each alternatives likely
outcomes Models should produce a probability distribution. NOT a
single forecast.@LMaccherone [email protected] 53. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Next time someone gives you reasons fortheir decision,
ignore them!Rather,ask them about the alternatives consideredandthe
models used to forecast the outcome ofthose
alternatives.@LMaccherone [email protected] 54. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)@LMaccherone [email protected] 55. Presenting Metrics
to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Monte Carlo ForecastingWhat it looks likeDefer to
Troysforecastingcontent to dothis.@LMaccherone [email protected]
56. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Seek tochange the nature ofthe
conversation@LMaccherone [email protected] 57. Presenting
Metrics to Get ResultsDial M for Metrics Seminar (Magennis and
Maccherone)Influencing with data@LMaccherone [email protected]
58. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)The rider and the elephant Direct the
rider Motivate the@LMaccherone [email protected] Shape
the pathJonathan HaidtThe Happiness Hypothesis(also mentioned in
Switch) 59. Presenting Metrics to Get ResultsDial M for Metrics
Seminar (Magennis and Maccherone)How to Influence People with
DataTop tips for influencing people with data Tell a good story
Become known for being right Avoid wars about semantics Imperfect
evidence is better than no evidence@LMaccherone
[email protected] Change the nature of the
conversation(remember the Monte Carlo burn chart forecast?)Larry
Maccheronehttp://www.businesscomputingworld.co.uk/top-tips-for-influencing-people-with-data/
60. Presenting Metrics to Get ResultsDial M for Metrics Seminar
(Magennis and Maccherone)Mediocrastan vs Extremestan@LMaccherone
[email protected] 100 people chosen at random Whats the average
weight Add to that person a 101st person, on the planent How much
does the average change Supposed you are were talking about wealth
How much does average change with adding BillGates. Author of black
swan. 61. Presenting Metrics to Get ResultsDial M for Metrics
Seminar (Magennis and Maccherone)Roadmap@LMaccherone
[email protected] 62. Presenting Metrics to Get ResultsDial M
for Metrics Seminar (Magennis and Maccherone)ALM Charts Assessment
& Tracking Performance IndexMonte Carlo Forecasting Survey
Design Benchmarking &@LMaccherone [email protected]
ModelWhat-If Analysis
TransformationPlanDevelopmentRecommendationEngineRoadmap
