Upload
colorado-morse
View
22
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Analytics and Witch Doctoring: A Cure for the Black Box Mentality. September 21, 2011 O’Reilly Strata Conference J.C. Herz [email protected]. Analytics: Occult Phenomenon. Very powerful Don’t understand it Practitioners possess arcane knowledge. Secret Algorithms. - PowerPoint PPT Presentation
Citation preview
Analytics and Witch Doctoring: A Cure for the Black Box
Mentality
September 21, 2011
O’Reilly Strata Conference
J.C. Herz
Analytics: Occult Phenomenon
QuickTime™ and a decompressor
are needed to see this picture.
Very powerful
Don’t understand it
Practitioners possess arcane knowledge
Greek Letters Are Your Kryptonite
QuickTime™ and a decompressor
are needed to see this picture. QuickTime™ and a decompressor
are needed to see this picture.
High Status Helplessness
• If you understood the technology, you’d be one of those people whose job it is to make technology work.
• You know, underlings
OODA Loop
Which of these does an analytic tool/technology do?
QuickTime™ and a decompressor
are needed to see this picture.
Critical Question: Data Quality
• How complete is it?
– Ex: 600 custom fields, only two have more than 50% coverage
• How accurate is it? How do you know?
• How consistent is it?
• Good test: make three calls to different parts of the company, to get an answer to a factual question that doesn’t require calculation.
Precision vs. Accuracy
When precision exceeds accuracy, you’re setting yourself up for analytic failure
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Critical Question: Half-Life of Data
• How long is the data accurate?
• How long is the data useful?
QuickTime™ and a decompressor
are needed to see this picture.
Critical Question: Tempo
• How fast will a decision be made on the basis of analytic insight?
• Quarterly?
• Daily?
• Within seconds?
• Milliseconds?
• Never?
• Realtime vs. Continuous vs. Batch
Critical Question: Real World Context
• Without real world data, “behavioral” metrics are misleading
• Where is the transactional data that validates insights from non-transactional data?
• How would you prove the magic analytics WRONG?
Critical Question: What is the Validation Test?
• Formulating the validation test keeps both the customer and the developer focused - and honest
• Suggest pay for performance, and see if the developer or vendor freaks out.
• Make sure validation is ongoing - in case the ground is shifting
Critical Questions: Workflow
• What workflow changes will this proposed capability require?
• People hate changing their workflow, even if it’s an improvement
• Never attribute to stupidity what can be attributed to laziness
• What is your plan for changing workflow? How do you enforce it?
Critical Question: Consequences
• What actions are you willing to take on the basis of validated analytic insight?– Change your product?– Change your marketing budget?– Change people’s job descriptions?– Re-allocate R&D budgets?
• What actions are you not willing to take?
Questions?
• J.C. Herz [email protected] (202) 213-3151
QuickTime™ and a decompressor
are needed to see this picture.
Technical Reality vs. Leadership Attention Span
QuickTime™ and a decompressor
are needed to see this picture.
Business Payoff vs. Intellectual Appeal
QuickTime™ and a decompressor
are needed to see this picture.
Social Network Analysis
Operations Research
Market Segmentation
Competitive Intelligence
Pilots to test new analyst tools with tiny amounts of generic data
360º Lead Scoring
Validate Marketing Effectiveness
Before You Rip ‘n’ Replace:
What is the exit cost of this technology?
Does “turnkey” mean monoculture?
Ugly Babies & Pretty Babies
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.