Driving Business Goals with Recommender Systems @ YAC/m 2015

Driving business goals with Recommender Systems

Konstantin Savenkov, COO Bookmate

[email protected], http://bookmate.com

Target audience

4

B2C Services

B2B Recom- mender Platforms

run a pilot estimate costs and benefits

determine fair price or scale to

start with

PROFIT

determine value for potential

clients run a pilot

set fair pricing model

5

Agenda

•  Recommender Systems: Academy, Technology, Business

•  Recommender Systems for content discovery

•  B2C Content Services: overview and business model

•  Driving business goals with Recommender Systems

•  customer acquisition cost •  lifetime value •  catalogue exploitation

•  Bookmate – E-Contenta case

6

RS

$ $ $ $

Agenda







7

RS

$ $ $ $

Academy vs. Tech vs. Business

8

How to improve performance by X%

How hard is to implement that?

A: T:

B: When gains match costs?

Evaluation of Recommender Systems

9

academy business offline evaluation

online evaluation

economic evaluation

•  user behavior history

•  RMSE •  MAP •  NDCG •  etc.

•  live users •  actual UX •  actual

inventory

•  NDCG •  CTR •  funnels •  response

time

•  live users •  actual UX •  actual

inventory •  business

model

•  CAC •  LTV •  COGS •  …PROFIT!

10

“It’s tempting, if the only tool you have is a hammer, to treat everything as a nail.”

* Recommender systems are cool, but they don’t substitute old good traffic quality, UX and pricing.

Abraham Maslow, The Psychology of Science, 1966

Agenda







11

RS

$ $ $ $

Recommender Systems for Content Discovery

•  preference elicitation

•  hard to describe preferences in a textual form

•  weak textual relevance

•  limited catalogue

12

“I WANT TO READ SOMETHING…”

EVEN FOR BOOKS!

LOOKING FOR UNKNOWN UNKNOWNS

REGIONAL SEGMENTATION

User with a book problem

13

Search case Recommendation case

Recommender Systems in the interface •  Any place in the interface, when number of objects to

show exceeds available space

•  Most of the interfaces are list-based

•  Hence, order and size of the list can be defined by either personalized or non-personalized algorithm

•  Explaining recommendations is a different topic

14

There is no “no recommender system” setting. If there’s “just something” or “popularity sorted”, that’s your RS. !

Bookmate Example

15

faceted filter

search results

book page

user library

notifi-cations

social feed

Agenda







16

RS

$ $ $ $

B2C Content Services

17

subscription, PPD or hybrid

limited attention and time

content may have different cost

Unit Economics

18

Business at scale (marginal revenue and expenses per user)

LTV

Cost of content

CAC

user

life

time

ARPU

ARPU

…

PROFIT!

How the product works

•  Each connection here is driven and improved by business activities

•  The content itself fits into a sort of a BCG matrix:

19

GROWTH

CO

STS

CAC

×

÷

Driving Business Goals

20

CAC

LTV

Content Costs

Marketing Expenses

New Customers

ARPU

Lifetime

Consumed Content Mix

Conversion

Retention

Reactivation

Exposed Content Mix

×

÷

Driving Business Goals

21

CAC

LTV

Content Costs

Marketing Expenses

New Customers

ARPU

Lifetime

Consumed Content Mix

Conversion

Retention

Reactivation

Exposed Content Mix

*

* the recommendation fairy

Agenda







22

RS

$ $ $ $

Option1: Improving conversion / САС

23

paywall

Option I: Improving conversion / CAC

Hypotheses to prove:

1. There’re enough users who will use RS output 2. Their conversion will be above average

A/B testing is the only way:

§  different channels convert with up to 20x difference §  current traffic mix is unpredictable and hard to control in case of app installs

Do pilots:

§  Run with limited resources, then extrapolate and decide if run full-scale

24

Option I: Improving conversion / CAC

Two approaches to estimate:

1. increase of revenue 2. decrease of CAC

Suits for estimating various models:

§  upfront costs (when the investments return)

§  flat fee (monthly license or added headcount)

§  variable costs (CPO or PaaS model)

25

UNIT ECONOMICS!

subscribers

marketing budget

Case Study: Bookmate + E-Contenta

Sounds promising!

Did 40% more users become converted?

Not really, as there was just 7% who didn’t know the book to start with.

26

Group A Group B

Decided to use this channel

Converted

3 starter books from editors

3 starter books from a cold-

start RS

2.17*X% X%

Y% 0.65*Y%

Overall conversion Z% 1.4*Z%

three-sigma

Let’s check the economics *

•  In case of using a third-party RS on a CPO basis, in this case the CPO is limited by $0.14 (actually, much less)

•  In case of a flat fee of $1000**/month, this is feasible starting from 7143 new subscribers/month, or $35K of marketing budget.

27 * CAC and marketing budget are model data ** some arbitrary number

1000 CAC = $5

Group A Group B

Blended conversion C% Blended conversion 1.028*C%

Increased conversion 1.4x for 7% of users

CAC = $4.86 +28

Blended conversion across all channels is C%

$5000 of traffic

Option 2: Improving retention / LTV Hypotheses to prove:

1. User pays as long as he finds what to read 2. There’re enough users who will use RS output 3. This channel has a discoverability above average

Ideal experiment:

§  A/B, then count actual lifetime §  with lifetime close to year, it’s too long to wait

Solution:

§  do separate A/B for different user cohorts (new, 1 month old, 2 months old etc.) §  estimate significant change in month-to-month retention for each cohort

28

Model case: estimating LTV improvement Let’s assume Recommender System led to 0.5%-3%* increase of month-to-month retention (old cohorts / new cohorts), Group A estimated lifetime is 9 months*.

29 * model data provided for illustration

That’s an equivalent for: •  increase of the lifetime by 2.6

months for Group B •  increase of LTV by 29% for

Group B

The area between the curves is equal to # of additional ARPUs

Option 3: Better catalogue exploitation

30

Just to give an idea:

•  long-tail content in general is cheaper (niche, back-catalogue and public domain) •  driving user out of search already improves margins •  adding a recommender system really changes a balance •  once you have the data from the pilot, estimation is quite straightforward

Conclusions for B2C services

•  The simplest recommender system would likely give you 80% of all possible upside. If it doesn’t, the problem is most likely not in the algorithm.

•  If you want to go beyond, run a pilot to assess costs and benefits, then estimate if you have enough scale to afford the solution.

•  If you deal with a third-party Recommender System convince them to fair pricing (e.g. free period until you have enough scale).

•  And, again 31

Conclusions for B2B Recommender Platforms •  Based on amount of traffic, price of marketing budget you can estimate value of

your solution for potential customers.

•  Based on pilot integrations, you may either define a fair price point for a particular customer or develop PaaS-style tiered pricing model.

•  Doing just a UX-applicable Recommender Systems leaves you a quite tight margin between LTV and CAC+COGS. Better take on the full user acquisition vertical.

•  TEASER: Bookmate + E-Contenta 2.0: E-Contenta integrates with remarketing solution and provides traffic, not just recommendations.

32

33

Wanna 1 month of Bookmate

for free? Go

http://bookmate.com/code

Use “yacm2015”

(valid thru June 2015)

Data & Analytics

Driving Business Goals with Recommender Systems @ YAC/m 2015