Click here to load reader

Analytics Lessons Learnt

  • View

  • Download

Embed Size (px)

Text of Analytics Lessons Learnt

  • Analytics - Lessons Learnt

    Dr. Venkata PingaliApril 1, 2016

  • Basic Process

  • Conceptual Process Biz Analytics Team

    Data Engg

    Qtns, Context

    Data Req


    Model Results

    Story TellingAll three roles could be in a single team!

  • Process in RealityBiz Analytics Team

    Data Engg

    Qtns, Context

    Data Req


    Model Results

    Story Telling


  • Process in RealityBiz Analytics Team

    Data Engg

    Qtns, Context

    Data Req


    Model Results

    Story Telling


    "80% of ..companies strategic decision go haywire.. flawed data

  • Nature of Domain

  • Sense-making with Purpose

    Goal is impact - real change in the real world Not mathematical machoness Not blogs, presentations, etc.

    Model + Delivery = Impact Model - an approximation to real world

    Three levels - Question, Domain, Process Realworld has (unknown) complexity Not an end in itself

    Delivery - Facilitation of incremental change Multiple levels - Mindsets, technology, processes

    Closing loop is a reality check

  • An imperfect search process

    Imperfect questions, data, and process Complexity discovered over time Iterative refinement

    Laborious, error prone, and always incomplete Data preparation (60-80% of work) is error prone Questions -> Answers -> Questions

    Initial framing is just the beginning Story will reveal itself over time

    Design for uncertainty

  • Successful Analytics Shifts Power

    There are winners and losers Change is always painful Efficiencies have to come from somewhere

    Mostly through power to contradict Upsets conventional wisdom

    Sometimes through new paths forward

    Analytics is serious business

  • Trust is #1 requirement

    Change require trust in output (evidence and path forward) Gaining trust is hard work

    Delivered by what you do and how All the time and everything you do

    Integrity required through the entire lifecycle Data Process Interpretation

    Design for trust

  • Math is either correct or not

    Sense-making may be qualitative but data or transformations are not Every step is mathematical step

    Correct math is the basis for trust Process is laborious Work should not be trusted by default!

    Hidden transformations are risky Excel changes Filtering rules

    Mathematical indiscipline will be punished

  • Efficiency is #2 requirement

    Data science getting out of the lab environment Decision makers have realized that they could be wrong (often?)

    Need to be contradicted only once - happening frequently Now they are asking for input in all areas

    Sea change in last 4 years Growing combinations - #decisions x #scope x #frequency x #depth

    Growing much faster than people & process can cope

    Process efficiency is essential to scaling

  • Team Character determines Quality

    Fundamentally about collaborative reasoning under uncertainty Need a creative group of people

    Balanced skill along multiple dimensions Domain (technology, business, individual) Approach (model, experiment, field work) Engagement (presentation, tech delivery, ops)

    Balanced process Increased curiosity bandwidth will give people mastery, purpose

    Sense of purpose

    Look to build a strong team

  • Surviving the Insight Ladder

    Step 1 Wranging - Get to facts at summary level Step 2 Discovery - Frame initial questions & iterate to get to real

    questions Step 3 Relevance - Meaningful imprecise answers Step 4 Accuracy - Meaning precise answers Step 5 Robustness - Meaningful, precise, robust answers

    Continuously increase curiosity bandwidth

    Time spent here = Curiousity bandwidth

  • Business

  • Has to be shared organizational experience

    Mistakes are frequent Through the entire lifecycle

    Domain knowledge is discovered More important than math

    Make analytics a collective experience

  • Costs are front-loaded

    Data preparation/wranging Takes arbitrary amount of time Time/Effort ~ #elements ^^ 2

    Errors in model development and operation Data version updates Changes in narratives

    Budgeting and expectation setting should be realistic

  • Empathetic delivery

    Analytics has collateral damage People get fired, budgets are cut, new responsibilities get added

    Empathetic positioning and language Understand that everybody wants to do their job well People are not dumb

    Incremental actionables Show way forward in byte chunks

    Plan the delivery carefully

  • Analytics work is risky

    Over-hyped context Bigger, better examples everywhere - real or imagined

    Burden of expectations/magic from customer Things go wrong

    Underwhelming/no results, methodological issues, wrong data

    Crisis as a teaching moment Culture of learning, understanding and continuous refinement

    Enable team to take risks and have honest conversations

  • Individual

  • Dont be pygmalion

    Dont fall in love with data It is imperfect like everything else

    Even simple data is too rich You see what you want to see

    Be deeply skeptical Explore without judgment, detached

    Develop non-judgment curiousity

  • Extra

  • Decision-maker Questions

    1. Where did the numbers come from? (Correctness, Lineage)a. Assumption, models, datasets

    2. Is this an accident? Does it hold now? (Reproducibility, Retargetability)a. Model, dataset, and question revisions

    3. Can you get the results faster? (Efficiency)a. Time, effort, cost

    4. Can you also analyze X? (Extensibility) a. Different dataset, question

    5. Could we try X? (Dataset generation - synthetic and real)a. What if scenarios, field experiments