Embed Size (px)
Citation preview
Using data to inform product decisions
Why we do need data to inform product development
ProductTank Cape Town - 26 March ‘15
Imagine this ...
... or this
Outline
Why
Driven vs Informed
What
5 things to be mindful of
Why do we need data?
Why do we need data?
What data do we need?
It’s about asking the right questions, you fool!!
Data & the product lifecycle
What do we want?
Assumption: “Sitting on a tractor all day isn’t the best use of my time”
Assumption: My users want to spend less time on the tractor so that they can spend more time on other tasks
Question: Is there a market need for driverless tractors?
What do we want?
Hypothesis: We believe this is true if the users of our MVP spend 20% more time on the farm
Approaches: One Single Metric, prototypes, MVP, observations, market research, diary studies
What do we want?
What do we want?
How should it work?
How should it work?
Question: How can we encourage people to discover and configure multiple cars?
Assumption: People will be encouraged to explore multiple cars if they see nice images of cars similar to the one they have just configured
How should it work?Hypothesis: We believe that adding images will drive car discovery. We know this is true if there’s a 30% increase in the average number of cars configured per person by end of May ’15
Approaches: A/B and MVT, behavioural plan & KPIs, prototypes and usability testing Approaches: A/B and MVT, behavioural plan & KPIs, prototypes and usability testing
“Blank slate”
“Blank slate”
One Single Metric: Percentage of users per variant who configure another car
Design and sample size: Minimum of 200 conversions per page to reach “statistical significance”
BUT: I can’t learn everything through this experiment!
How is it working?
How is it working?
Question: Is our product / feature meeting the hypothesis?
Assumption: We believe that this feature will be used by 50% of our first time car buyers in the UK within the first month after release
Question: What is our strongest market or user segment?
How is it working?Hypothesis: We know that our assumption is correct if we see a 20% increase (on the current benchmark) in the number of UK first time car buyers purchasing a car through our site
Approaches: Usage tracking, user testing, product retrospectives and refine or reject hypothesis
Identify opportunities for product improvement or reasons for discontinuation
How is it working?
Gathering the right data
What can quant data tell us?
What can quant data tell us?
Analytics
What can qual data tell us?Qualitative data can help us: !
Understand the why behind quantitative data !
Get insight into what people think and feel !
Learn about a product idea or prototype !
!
What can qual data tell us?
What can qual data tell us?
What can qual data tell us?
Data driven
Data drivenA/B or multi-variate test continuously !
Focus on the “One Metric That Matters” !
Build hypothesis around key KPI !
Optimise your product based on data !
Are we making a noticeable difference?
BUT... What data cannot tellIs it a good product idea? !
Metrics do not always offer you the full picture !
Data is one of the factors that feed into a decision !
We typically do not own all product decisions
Data informed
Data informed
Data
Users
Intuition
Competition
Technology
Brand
Strategy
BusinessRegulation
Time
Data informedData is one of the factors to consider !
Focus on the questions that you want answered !
You cannot replace intuition or creative ideas with data !
Assess impact on relevant areas
5 things to be mindful ofFocus on asking the right questions !
Data can’t replace intuition !
Be clear on hypothesis, sample size and timings !
Build and launch with data in mind !
Listen to the data and act accordingly!
SO ...
Embrace the data, don’t fear it!
Related linkshttp://svpg.com/assessing-product-opportunities/ !http://www.romanpichler.com/blog/goal-oriented-agile-product-roadmap/
http://vimeo.com/14999991
http://www.realityisagame.com/archives/390/wooga-follows-zynga-in-metrics-driven-game-design/
http://marcabraham.wordpress.com/2013/05/03/book-review-lean-analytics/
http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-targets/
http://marcabraham.wordpress.com/2013/09/09/some-considerations-regarding-data-driven-design/
http://insideintercom.io/the-problem-with-data-driven-decisions/
http://www.webdesignerdepot.com/2013/05/the-perils-of-ab-testing/
http://andrewchen.co/2008/09/08/how-to-measure-if-users-love-your-product-using-cohorts-and-revisit-rates/
http://codeascraft.com/2012/06/21/building-websites-with-science/
Related links!
https://marcabraham.wordpress.com/2015/03/05/what-is-guerrilla-testing/
http://www.slideshare.net/LilyDart/guerrilla-testing-for-content
https://marcabraham.wordpress.com/2015/02/14/learning-more-about-running-ab-tests/
https://marcabraham.wordpress.com/2015/02/04/book-review-thinking-with-data/
https://marcabraham.wordpress.com/2015/01/26/book-review-designing-for-behavior-change/
http://www.simplypsychology.org/qualitative-quantitative.html
http://data.heapanalytics.com/dont-stop-your-ab-tests-part-way-through/
https://marcabraham.wordpress.com/2014/12/22/book-review-web-metrics/
!
!
!
