Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
SLIDE NO. 1
TAKE CONTROL OVER YOUR KPIsHilda KosorusSenior Data Scientist
SLIDE NO. 2
About me
2017
Runtastic: Senior data scientist
2015
Runtastic:Data scientist
2010
JKU: Research assistant
Academia Industry
2013
JKU:Senior researcher
SLIDE NO. 3
ABOUT RUNTASTIC
SLIDE NO. 4
4We are
founders
8We are
years old20
We were profitable after just
months
235We are
employees
40+We come from
countries
3We have
offices in Linz, Viennaand Salzburg
15Our products are
available in
languages
1We are
team with a shared vision
RUNTASTIC BY THE NUMBERS
SLIDE NO. 5
SLIDE NO. 6
SLIDE NO. 7
EACH DAY WE HAVE
150,000NEW DOWNLOADS
WE HAVE MORE THAN
130MREGISTERED USERS
WE BOAST OVER
246MTOTAL APP DOWNLOADS
WE HAVE MORE THAN
4.8MFANS AND FOLLOWERS
TAKE CONTROL OVER YOUR KPIsAdvice from a Data Scientist
PRESENTATION TITLESLIDE NO. 9
WHAT and WHY?
PRESENTATION TITLESLIDE NO. 10
What are KPIs?
KPI = Key Performance Indicator
“a quantifiable measure used to evaluate the success of a business”
PRESENTATION TITLESLIDE NO. 11
Why are KPIs useful?
measurable / quantifiable
data-driven
gauge the effectiveness of processes
business health indicators
setting goals
common language → common goal
PRESENTATION TITLESLIDE NO. 12
Examples
● Number of newly acquired customers● Revenue● ROI● Retention/churn rate● Conversion rate● NPS● Sales growth
PRESENTATION TITLESLIDE NO. 13
HOW: 7 STEPS
PRESENTATION TITLESLIDE NO. 14
1. CHOOSE WISELY
PRESENTATION TITLESLIDE NO. 15
Choose SMART
● Specific - has a clear purpose for the business
● Measurable
● Achievable
● Relevant to the success of the organization
● Time phased - the value represents a predefined and relevant period
PRESENTATION TITLESLIDE NO. 16
You advise, management decides
● Be proactive● Do your homework● Communicate & consult● Make it work either way
PRESENTATION TITLESLIDE NO. 17
2. UNDERSTAND THOROUGHLY
PRESENTATION TITLESLIDE NO. 18
Why should you care?
● You can’t optimize what you don’t understand
● Won’t know why they perform good/bad
● Can’t act upon them effectively
● Waste of resources → invested in the wrong place for the wrong reasons
● Most probably won’t reach your goals
PRESENTATION TITLESLIDE NO. 19
What should you do?
1. Where is the data coming from? ● Is the source reliable?
2. How is the metric calculated?3. Analyze
● components, trend, seasonality, edge cases● relationships to other company metrics & drivers● at different granularity levels
4. Monitor● seasonality effects, patterns, outliers
PRESENTATION TITLESLIDE NO. 20
Focus on the basic components
Almost all KPIs are aggregates
● Count of non-repeatable events○ without churn (e.g. # registered users)○ with churn (# active subscribers)
● Windowed aggregates (e.g. # monthly active users)● Ratio (e.g. % conversion rate)
PRESENTATION TITLESLIDE NO. 21
What a DS should see:
Example: # active subscribers
new
churn
What a managers sees:
PRESENTATION TITLESLIDE NO. 22
Monitor
PRESENTATION TITLESLIDE NO. 23
3. ESTIMATE FUTURE BEHAVIOR
PRESENTATION TITLESLIDE NO. 24
After all, they are just indicators...
● KPIs are similar to measuring instruments (e.g. thermometer)● useful to know past/current state (temperature increased or decreased)
It is more critical to know what opportunities and dangers lie ahead.
Drivers are your most valuable allies.
PRESENTATION TITLESLIDE NO. 25
How to predict?
● Forecasting methods○ growth-based○ curve fitting (linear, polynomial, regression methods, etc.)○ probabilistic models (survival analysis, HMM)○ time series forecasting (ARIMA)○ machine learning (e.g. classification)
● A plethora of libraries (R, python)
PRESENTATION TITLESLIDE NO. 26
Time series forecasting
● developed by Facebook DS team● open source● python & R● facebook.github.io/prophet
PRESENTATION TITLESLIDE NO. 27
How does it work?
● decomposable time series model○ trend, seasonality, holiday
○ models non-period changes○ represents periodic change○ represents effects of irregular
events● interpretable parameters
PRESENTATION TITLESLIDE NO. 28
Capabilities
● Growth (linear, logistic)● Changepoints● Seasonality components
○ yearly, monthly, weekly, daily○ additive, multiplicative
● Holidays○ past and future dates○ lower and upper window of effect
● Deals with missing values● Deals with irregularly spaced data● Diagnostics● Visualizations
PRESENTATION TITLESLIDE NO. 29
Why prophet?
● automatic forecasting techniques○ hard to tune○ too inflexible to incorporate useful assumptions or heuristics
● prophet○ flexible, easy to use○ fast○ interpretable○ it works!
PRESENTATION TITLESLIDE NO. 30
Example in R
PRESENTATION TITLESLIDE NO. 31
4. AUTOMATE
PRESENTATION TITLESLIDE NO. 32
ETL
Reports
R + T-SQL
Data sources
Interactive dashboard
Deployment infrastructure
PRESENTATION TITLESLIDE NO. 33
1
2
R and Transact-SQL
PRESENTATION TITLESLIDE NO. 34
Learnings
● Communication & collaboration○ Define clear responsibilities (DE, DS)
● Efficient deployment○ Define process○ Empower DS to deploy independently○ Code reusability○ Standardize forecasting data products
● Platform stability○ R launchpad management on SQL server
PRESENTATION TITLESLIDE NO. 35
5. INSPIRE TRUST
PRESENTATION TITLESLIDE NO. 36
Evaluate for model selection
● Historical data for training● Parameter selection
○ changepoint prior○ seasonality prior○ growth method○ etc.
● Cross-validation● Evaluation metrics (e.g. MAPE)● Visualization approaches
PRESENTATION TITLESLIDE NO. 37
Evaluate past predictions
● How accurate were our past predictions?● Monitoring● Sanity checks
○ trend changes○ seasonal changes○ other interferences
● Re-evaluate parameter settings
PRESENTATION TITLESLIDE NO. 38
6. ACT
PRESENTATION TITLESLIDE NO. 39
Making use of predictions
● Help understand trends and seasonality● Defining goals
○ realistic○ ambitious
● Detect trend changes early on○ automate
● Derive meaningful actions○ resource allocation○ identify measures with high impact
PRESENTATION TITLESLIDE NO. 40
7. OPTIMIZE
PRESENTATION TITLESLIDE NO. 41
Good enough, but not great
● Granularity○ Break down metric for consistent behaviour ○ Segments (e.g. product, platform, acquisition source)○ Components (e.g. new and churned customers)
● Events -- non-organic interferences (e.g. ads, campaigns, offers)○ Identify effect○ Collect data & consider future occurrences○ Plan & optimize
● Parameters○ Monitor: identify potential improvements○ Re-evaluate regularly○ Future work: automate
PRESENTATION TITLESLIDE NO. 42
Example: using holidays
PRESENTATION TITLESLIDE NO. 43
To sum up...
1. Choose wisely2. Understand thoroughly3. Monitoring is not enough4. Automate5. Inspire trust6. Act7. Optimize continuously
PRESENTATION TITLESLIDE NO. 44
SLIDE NO. 45
THANK YOURuntastic.com
Place photo over me
Hilda KosorusSenior Data Scientist
SLIDE NO. 46
We are hiringDATA ENGINEERS