Embed Size (px)
Citation preview
UPCOMING TALKSWhat to do with data from 1 billion smart devices in China?Peter Warman - CEO & Co-founder, Newzoo From Data Science to Data Impact: On many ways to segment your players & more Volodymyr (Vlad) Kazantsev - Head of Data Science, Product Madness Trends in game analytics: What’s happening (and why)?Heather Stark - Analyst, Kirnan Limited
35 2015 © All rights reserved to
From Data Science to Data Impact:
On many ways to segment your players
Volodymyr (Vlad) KazantsevHead of Data Science at Product Madness
37
Heart of Vegas in (public) Numbers
iPad US - #13 top grossingiPhone US - #32 top grossingAndroid - #44 top grossing
US (games) AustraliaiPad - #1 top grossingiPhone - #1 top grossingAndroid -#3 top grossing
[email protected] volodymyrk
38
Data Impact Team
Ad-hoc analytics; dashboards
Deep dive analysis; Predictive analytics
ETL, R&D
[email protected] volodymyrk
39
Data Impact Team
Insights
Data
Science
Data
Engineering
7 people; 4 in London office
[email protected] volodymyrk
Ad-hoc analytics; dashboards
Deep dive analysis; Predictive analytics
ETL, R&D
40
Ad-hoc analytics; dashboards
Deep dive analysis; Predictive analytics
ETL, R&D
Data Impact Team
Insights
Data
Science
Data
Engineering
7 people; 4 in London office
We Are Hiring [email protected]
[email protected] volodymyrk
41
Technology Stack
ETL orchestration
Transformation& Aggregation
SQL
Data Products
Reports
Dashboards
+
42
few examples ..
A B
A/B TestsCustomer Lifetime Value
days
$ va
lue
Segmentation
group 1 group 2 group 3 group 4
43
Segmentation Basics1
44
Successful segmentation is the product of a detailed understanding of your market and will therefore take time
- Market Segmentation: How to Do it and Profit from it, 4th edition: Malcolm McDonald
45
BasicsCustomers have different needs and meansSegmentation can help to understand those differencesWhich can help to deliver on those needsAnd drive higher profitability
46
What is a Player Segment?
A segment is a group of customers who display similarities to each other...
Customers move in and out of segments over time
47
How many segments are there?
There is no one right way to segment (not should there be):
Many different approaches and techniques
Mix of art, science, common sense, experience and practical
knowledge
Depends on business needs and availability of data
Don’t aim to build one holistic model to meet all needs
48
Strategic Management
Product Development
Marketing Operations
Comments
Geography /Demographics
Loyalty / Length of Relationship
Behavioural
Needs-based
Value Based
49
Strategic Management
Product Development
Marketing Operations
Comments
Geography /Demographics
✭✭ ✭✭ ✭✭Separates players by country, city, city-district, distance from land-based casinos. By generational profile: boomers, Gen-Y, Gen-X.
Loyalty / Length of Relationship
✭✭✭ ✭ ✭✭✭ New players, on-boarding, engaged, lapsed, re-engaged, cross-promoted.
Behavioural ✭ ✭✭✭ ✭✭✭Based on identifying player’s behaviour characteristics that help to understand why customer behave the way they do
Needs-based ✭ ✭✭✭ ✭ Divide customers based on needs which are being fulfilled by playing Online Slots
Value Based ✭✭✭ ✭ ✭✭ Based on present and future value of the customer (RFM/CLV)
50
Land-based Slots Player segmentation
<10%
>50%
51
Segmentation = building a taxonomyAll Players
New(<28 days)
Established (>28d)
Payer Non Payer0-2 days 3-7d 8-27
<30 spins >30 … High V Med V Low V Engaged Casual…VIP Concierge
52
..and simplifying it daily useAll Players
New(<28 days)
Established (>28d)
Payer Non Payer0-2 days 3-7d 8-27
<30 spins >30 … High V Med V Low V Casual…
New High Value Med Value Low Value Engaged Casual
Engaged
53
How to profit from Segmentation?2
54
Clients of Segmentation
○ Strategy and Finance
○ Product development
○ Marketing operations
55
Strategy and FinanceThis Month
high-value med-value low-value super free-rider casual slotter recently lapsed
high-value 55.27% 30.06% 4.81% 5.54% 2.00% 2.32%
med-value 11.11% 42.50% 25.25% 10.92% 6.20% 4.02%
low-value 0.59% 7.72% 36.02% 30.59% 17.12% 7.96%
super free-rider 0.04% 0.30% 2.76% 70.50% 22.22% 4.18%
casual slotter 0.01% 0.10% 0.96% 8.98% 51.37% 38.58%
recently lapsed 0.05% 0.22% 1.01% 8.93% 13.00% n/a
New 0.01% 0.08% 0.67% 3.22% 31.05% 64.97%
This Month 0.15% 0.54% 2.13% 21.56% 31.22% 23.03%
Last Month 0.11% 0.43% 2.03% 21.09% 37.19% 27.20%
Last
Mon
th
56
Strategy and FinanceThis Month
high-value med-value low-value super free-rider casual slotter recently lapsed
high-value 6.80% -0.45% -1.66% -2.39% -1.07% -1.24%
med-value 3.09% 2.60% -2.81% -2.12% -0.60% -0.16%
low-value 0.11% 0.90% -1.63% 1.99% -0.54% -0.82%
super free-rider 0.01% 0.05% -0.05% -2.05% 2.58% -0.54%
casual slotter 0.00% 0.02% 0.05% -1.26% 2.71% -1.54%
recently lapsed -0.01% -0.05% -0.35% -4.21% -8.43% N/A
New 0.01% 0.04% 0.36% 1.59% 16.17% 1.21%
Manage transitions, not churn
Last
Mon
th
57
Product Development
New Slot Game Released
Coins Spent
58
Product Development
Geo: AustraliaValue: Low-valueBehaviour: Prefer Medium bet
New Slot Game Released
Coins Spent
59
Marketing
Objective Behavioral RFM/CLV geo/demographic Lifecycle
Sale Events
Monetization campaigns
Retention campaigns
Re-engagement
VIP management
60
How to actually do segmentation?3
61
Pillars of Successful Segmentation Project
Business knowledge
Data knowledge
Analytical skillsPeople
Process
Technology
ETL
Machine Learning
Business Intelligence
Product Integration
Marketing
Product
Data Services
62
Top-down approach to segmentation
1. Define objectives and therefore customer characteristicsa. dd
2. Choice method to split usersa. d
3. Prioritise segments to targeta. d
4. Operationalise segmentationa. s
5. ‘land’ the segmentation within the organization
63
Bottom-up approach360o player
view
Segmentation
Player transitions
Tailored interventions
Prioritisation and testing
● Build database to provide 360o view of the customer● Demographic, behavioural, payments, etc.● Add predictive attributes, such as conversion probability, churn risk, LTV, etc.
● Segment customers by desired attributes: more than one approach● Use robust statistical techniques for clustering or validation of empirical segmentation● Ensure segmentation is intuitive for the business and can be used across business functions
● Identify how players are moving from one segment to another (segment transition matrix)● Determine value levers and identify potential improvement ideas
● Create tailored interventions (CRM, push ..), aimed at moving customers to more valuable segments● Build predictive models to detect best offer and prevent undesirable transitions
● Prioritise interventions based on expected LTV uplift and ease of implementation● Test interventions through experimentation
64
How to actually do segmentation?
Just Look at Data Clustering Decision Trees
Player Attributes
de-correlate
Normalise Scale
65
de-correlate and normalise
Player 1 more similar to Player 2 ?Player 3 more similar to Player 2 ?
Weekly Play Summary
66
de-correlate and normaliseWeekly Play Summary
67
de-correlate and normalise
Player 1 more similar to Player 2 !
68
de-correlate and normalise
Player 1 more similar to Player 2 !
69
de-correlate and normalise
Player 1 more similar to Player 2, isn’t he?
70
de-correlate and normalise
Player 3 more similar to Player 2 !
71
What now?
K-meansHierarchical ClusteringDecision Trees.. and many more
72
Decision Tree for ClusteringAll Payers
500 (next month>$100): 4.7%10000 did not: 95.3%
Last_months_dollars <=$22 (next month>$100): 0.04%
5000 did not: 99%
Last_months_dollars >$2498 (next month>$100) > $100: 9%
5000 did not: 91%
Transactions <=10243 (next month>$100): 5.5%
4200 did not: 94.5%
Transactions > 10255 (next month>$100): 24%
800 did not: 76%
73
Decision Tree for ClusteringAll Payers
500 (next month>$100): 4.7%10000 did not: 95.3%
Last_months_dollars <=$22 (next month>$100): 0.04%
5000 did not: 99%
Last_months_dollars >$2498 (next month>$100) > $100: 9%
5000 did not: 91%
Transactions <=10243 (next month>$100): 5.5%
4200 did not: 94.5%
Transactions > 10255 (next month>$100): 24%
800 did not: 76%
Low Value
Medium ValueHigh Value
74
Segmentation at Product Madness4
75
Lifestage Segmentation
On-Boarding
Disengaged
Engaged
not played game
Churned
76
On-boarding segment
On-Boarding
Disengaged
Engaged
not played game
Churned
77
On-boarding segment
On-Boarding
Disengaged
Engaged
not played game
Churned
78
Lifestage Segmentation
On-Boarding
Disengaged
Engaged low riskhigh risk
low riskhigh risk
low riskhigh risk
not played game
churned
churned
Churned
79
Behavioural Segmentation
Average BetGifts per DayBonuses per DayMachine StickinessDays PlayedSpins per DayPreference for New Machines%% of spin on High-Roller machinesBig Win Stickinessetc.
Hierarchical Clustering
80
Behavioural Segmentation
81
Infrastructure
Data Warehouse
Segmentation Engine
CRM Email GAME Reporting Ad Hoc Analytics
Predictive Analytics
82
Segmentation for A/B tests
A B
83
Segmentation for A/B tests
A B
84
Bonferroni correction:
Bayesian Hierarchical Model
Combine stats with Market Intuition!
Adjustment for multiple testing
𝛼adjustted = 𝛼desired/M
Kinran@
HASt
ark
What’s going on?
and why
Kinran@
HASt
ark
Market forces
Tech enablers
Frontier zones
Failure prevention
Kinran@
HASt
ark
Kawanabe Kyosai
Kinran@
HASt
ark
Market forces
Kinran@
HASt
ark
Kinran@
HASt
ark
...[according to VB research] most mobile-first companies are trying to pay between $1 and $1.50 for users, but they are only getting quality users at multiples of those numbers...
August 12 2015 http://venturebeat.com/2015/08/12/this-service-tells-you-what-supercell-machine-zone-and-other-big-publishers-spend-on-user-acquisition/
Kinran@
HASt
ark
...[according to AppScotch] Machine Zone is currently spending somewhere around $12 per user with AdColony, InMobi, and Unity Ads, up to $20 per user with Vungle, and between $2 and $30 per user with Chartboost...
August 12 2015 http://venturebeat.com/2015/08/12/this-service-tells-you-what-supercell-machine-zone-and-other-big-publishers-spend-on-user-acquisition/
Kinran@
HASt
ark
LTV modelling
attributionprediction
Kinran@
HASt
ark
Engagement insight
play patternspend pattern
Kinran@
HASt
ark
Kinran@
HASt
ark
Tech enablers
Kinran@
HASt
ark
Kinran@
HASt
ark
Kevin Schmidt and Luis Vicente, Mind CandyPractical real-time approximations using Spark Streaming
hyperloglogs to count uniquesbloom filters to count revenuestream-summary for top-k
(metwally agrawal abbadi 2005)
from nucl.ai Data Science track (to be published)earlier version from huguk available nowhttp://www.slideshare.net/huguk/fast-perfect-practical-realtime-approximations-using-spark-streaming
Kinran@
HASt
ark
Image: re-work.co Deep Learning Summit Boston
Kinran@
HASt
ark
http://ufldl.stanford.edu/wiki/index.php/Visualizing_a_Trained_Autoencoder
Kinran@
HASt
ark
https://www.flickr.com/photos/darronb/6502473781
Kinran@
HASt
ark
Frontier zones
Kinran@
HASt
ark
Kinran@
HASt
ark
Miloš Milošević, Nordeus
Early Churn Prediction and Personalised Interventions In Top 11later detection is more accurate - but less usefultried many techniques – logistic regression good!cluster users based on first day gameplaycustomise messaging based on clustersincreased D1 retention (and downstream metrics)
from nucl.ai 2015 Data Science track (to be published)writeup available on gamasutra now:http://www.gamasutra.com/blogs/MilosMilosevic/20150811/250913/How_data_scientists_slashed_early_churn_in_Top_Eleven.php
Kinran@
HASt
ark
http://mba-suits.com/first-set-pink-white-grey-suit/http://www.ahipster.com/hipster-fashion-knitted-jumpers/hipster-clothes-wooly/
Kinran@
HASt
ark
Failure prevention
Kinran@
HASt
ark
Meta S. Brown Analytics failure and how to avoid it
Analytics programs fail....... because they lack a viable plan for success
Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it
Kinran@
HASt
ark
Meta S. Brown Analytics failure and how to avoid it
Analytics programs fail....... because they lack a viable plan for success
Define success, and who decides on it
Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it
Kinran@
HASt
ark
Meta S. Brown Analytics failure and how to avoid it
Start with a business problema small oneunderstand the business problem really well
As you scale up – pay attention to process replicable! replicable! replicable!
Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it
Kinran@
HASt
ark
Meta S. Brown Analytics failure and how to avoid it
Use only as much data as you need to
The best use case for Big is personalisation
Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it
Kinran@
HASt
ark
What next?
JOIN IN THE CONVERSATION PARTICIPATE IN THE NEXT GIAF
Analytics for Games www.deltadna.com/giaf
#UKGIAF
