MetricsJanuary 2015

Niko Vuokko, Sharper Shape

What are metrics ?

Metrics are the eyes of the business

Eyes are for seeing where you’re stepping and where you want to go

Metrics are not for looking cool on the lobby screen

Which metrics should I follow ?

You don’t pick metrics, you pick business problems

Visible change in a metric visible change in the business

Business problems change and evolve

Seeing problems is not enough

Metrics should point out the root cause and hint at the solution

Example: New subscription-based app

Most effective user acquisition channel ?

Most efficient organic growth mechanism ?

How to fix onboarding ?

What features are unused ?

Should we make a ”special offer” after 2 or 5 days ?

Example: Older IAP-based app

Where are under-penetrated segments remaining ?

What makes users leave ?

What type of content drives monetization ?

Is there content saturation ?

What is my problem ?

User acquisition: example metrics

New users

Active users

Magnet features

Acquisition cost, per channel, country, user revenue, etc.

Channel traffic quality (this is tricky)

Engagement: example metrics

Back in X days after first use

Session length and its relation to revenue/retention

Feature coverage and popularity

Funnels, onboarding effectiveness

Retention: example metrics

It’s way cheaper to keep a user than to find a new one

Active after X days since first use

Time between visits

Weekly churn

Core features, what keeps users coming back?

Monetization: example metrics

Most freemium apps get a 2 % monetization rate

Monetizing features, what kind to introduce next?

Content saturation, i.e., spending walls

Promotion success, which hooks work?

Time of first monetization

To action

Treat users as ”somewhat” individual

Analysis and optimization across the whole userbase is not worth it

Analysis and optimization of individual users is not worth it

Find criteria that produce noticeable differences between groups

This may vary from metric to metric

Subgroup examples

”Impact of app localization varies wildly between countries”

”Users who installed during a weekend can be converted more

aggressively”

”Users with an animal avatar react great to this promotion”

”Launching the new version made user count go up, but

conversion rates suffered”

”Feature X is very popular in average, but very little among

paying users”

Practical issues with metrics

Data quality is absolutely horrible in many cases

Special doom pits: timestamps, IDs

The product and the users change => data changes

Long term aggregates go wrong

Metrics lose their meaning

Statistical significance

Humans are by nature horrible at interpreting statistics

Things get even worse when lots of data and no clear goal

You are not an exception

Guidelines

Be wary of any signals other than the painfully obvious ones

Always verify

Even service providers screw up multiple hypothesis testing

Service providers vs. DIY

Collecting and analyzing is expensive to a small team =>stay with service providers until you can’t

Decent services: GameAnalytics, Omniata, MixPanel, KissMetrics

Collect as much as you can, the use cases will emerge

Your data is almost certainly tiny => don’t overdo the tools

Getting data collection right MUCH more difficult than you expect

Getting the numbers right is MUCH more difficult than you expect

Power lawsThe new normal

Things are not normal

School teaches you that everything is a Gaussian

That’s just not true

Most things follow a power law, not a normal distribution

People don’t act the way you think

This is what most revenue/engagement/whatever metrics look like

Next, remove the non-paying users

But the result will not be like this normal distribution

This is the actual form

The numbers are highly concentrated and go pretty high

The curve follows the power law

Log axes produce a straight line

Another example

Number of users

Revenue per user

Power law

Follows from principle: “Whoever has will be given more”

Example: Web pages get links in proportion to their popularity

=> virtuous cycle

Characterized by 1) huge whales 2) huge mass at the bottom

Implications of power laws

Averages are worse than useless

Your userbase has very diverse subsets, treat them that way

More users means more users in the future

(App store ”Featured” actually works)

=> Only two relevant factors: new users and especially retention

Network effects are very powerful

Thank you!

REMEMBER!

You’re solving business problems, NOT watching cool charts