From a toolkit of recommendation algorithms into a real business:

13.09.2012.

the Gravity R&D experience

The kick-start

13.09.2012.2 From a toolkit of recommendation algorithms into a real business

Facing with real needs

What we had• rating prediction algorithms

• coded in various languages

• blending mechanism

• accuracy oriented

What clients wanted• recommendations that

bring revenue

• robustness

• low response time

• easy integration

• reporting

13.09.2012.3 From a toolkit of recommendation algorithms into a real business

What we do?

13.09.2012.4 From a toolkit of recommendation algorithms into a real business

users

content of service provider

recommender

Explicit vs implicit feedback

No ratings but interactions

sparse vs. dense matrix

requires different learning

13.09.2012.5 From a toolkit of recommendation algorithms into a real business

Increase revenue: A/B tests

against the original solution

internally

13.09.2012.6 From a toolkit of recommendation algorithms into a real business

Robustness

13.09.2012.7 From a toolkit of recommendation algorithms into a real business

IMPRESSApplication Server #1

Reco LAN

IMPRESSApplication Server #2

Reporting Subsystem

Platform OSS/BSS

Database #1 Database #2

Backend LAN

SOAP

CSV over FTP

Firewall

IMPRESS Frontendweb server #1

TV Service LAN

End users

Management LAN

Nagios MonitoringAggregator

IMPRESS Frontendweb server #2

Load Balancer

HP OpenView

SQL SQL

HTTP/ SQL

HTTP/ SQL

HTTP HTTP(S)

SNMP

Time requirements

• Response time: few ms (max 200)

• Training time: maximum few hours

• regular retraining

• incremental training

• Newsletters:

• nightly batch run

13.09.2012.8 From a toolkit of recommendation algorithms into a real business

Productization

13.09.2012.9 From a toolkit of recommendation algorithms into a real business

IMPRESSfor

IPTV, CATV and satellite

Recommends

Personally Relevant

Linear TV, VOD, catch-up TV and more

Gravity personalization platform

AD•APTfor

ad networks and ad server providers

Recommends Personally Relevant

ads

RECOfor

e-commerce

Recommends Personally Relevant

products & services

The 5% question – Importance of UI

Francisco Martin (Strands): „the algorithm is only 5% in the success of the recommender system”

• placement below or above the fold scrolling easy to recognize floating in

• title not misleading explanation like

• widget carrousel static

13.09.2012.10 From a toolkit of recommendation algorithms into a real business

Recommendation scenario

13.09.2012.11 From a toolkit of recommendation algorithms into a real business

Item2Item recommendation

logic: the ad’s profile will be

matched to the profile model of

available ads

Marketing channels

13.09.2012.12 From a toolkit of recommendation algorithms into a real business

Changing the order of two boxes: 25% CTR increase

Cannibalization

• Goal: increase user engagement

• Measurements

• average visit length

• average page views

• Effect of accurate recommendations:

• use of listing page ↓

• use of item page ↑

• Overall page view: remains the same

• Secondary measurements

• Contacting

• CTR increase

13.09.2012.13 From a toolkit of recommendation algorithms into a real business

?

Evolution: increased user engagement

• not a cold start problem

• parameter optimization and user engagement

13.09.2012.14 From a toolkit of recommendation algorithms into a real business

KPIs – may change during testing

13.09.2012.15 From a toolkit of recommendation algorithms into a real business

Complete personalization: coupon-world

• Newsletter (daily + occassionally)

• Ranking all offers on the website

• top1 item

• category preferences

• user metadata (gender, age, …)

• user category preferences (seldom given)

• item metadata

• context

• customer vs. vendor

13.09.2012.16 From a toolkit of recommendation algorithms into a real business

Business rules – driving/overriding ranking

13.09.2012.17 From a toolkit of recommendation algorithms into a real business

Contexts

13.09.2012.18 From a toolkit of recommendation algorithms into a real business

Context at TV program recommendation

• TV (EPG program & video-on-demand) explicit and implicit identification of the user in the household time-dependent recommendation

13.09.2012.19 From a toolkit of recommendation algorithms into a real business

Some results (offline)

13.09.2012.20 From a toolkit of recommendation algorithms into a real business

Recall@20 MAP@20 Recall@20 MAP@20Grocery 64,31% 137,96% 89,99% 199,82%TV1 14,77% 43,80% 28,66% 85,33%TV2 -7,94% 10,69% 7,77% 14,15%LastFM 96,10% 116,54% 40,98% 254,62%

Dataset

Improvement using seasoniTALS iTALSx

Recall@20 MAP@20 Recall@20 MAP@20Grocery 84,48% 104,13% 108,83% 122,24%TV1 36,15% 55,07% 26,14% 29,93%

Dataset

Improvement using SeqiTALS iTALSx

Some results (online)

Anecdotes

• Item2item recommendations – bookstore

• Placebo effect

• buyer vs. seller

13.09.2012.21 From a toolkit of recommendation algorithms into a real business

Conclusion

• Offline and online testing

• From simple to sophisticated

• Many more potential fields of application

13.09.2012.22 From a toolkit of recommendation algorithms into a real business