Analytical factory in a CRM contextHans de Wit, Senior Data Scientist, Telenor Norway

My passion: Making the unreal happen

My key goal:

Hans de Wit

• Telenor Mobile Norway (since 2013)• Advanced Analytics & Data Science

Manager• ING Bank, The Netherlands

• Senior member 'model‘/Innovation-team ING Retail Customer Intelligence

• Member analytical campaign management ING Bank Customer Intelligence department, 1997-2005

• ING Card, 2005-2008• Direct Marketing, Credit Risk, Fraud

• Master of Marketing (SRM) and bachelor of Commercial economics and Direct Marketing.

2

Sample path to traditionalcampaigns wih typically averageresponse rates

From one-offs to real-time: What makes the difference?

Dep

th o

fMar

ketin

gIn

sigh

t

One_off(Manual

Degree of Marketing Automation

Repeatable(Manual)

Scheduled(Automated)

Event-based(automated)

Real time(automated))

Ad HocLists

Profiling &Segmentation

PreditiveModelling

Detect Changes in Behavioral pattern

ContactOptimalization

Niche AutomatedCustomer Relevancy

First generation Spam

Sample path to optimizedrelevancy and timeliness Sample path to failed marketing3

An overview of what is “under the hood”

4

Life Cycle of a Model = Model FactoryIdentity

business problem

Data preparation

Data exploration

Transform & select

Analyticalmodeling

Validatemodels

Deploymodels

Evaluate/monitor results

5

Evolution of Data Mining Processes

Old Data Mining Process• run a simple process

• One person responsible for all.

• build more sophisticated and powerful models.

New Data Mining Process = Model Factory• speed up the computation speed

• and administer the entire process

6 6

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Reduce time-to-market

7

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Identify business & problem

8

• New campaign or New product.• I have problems that need solving…

• I don’t know which are my good customers!• Many of my customers are leaving!• I don’t know what I can say to them to avoid it!

• Business• Inbound = AST Controller• Outbound = Campaign manager or direct to marketing manager.

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Often 80% of time spent is on data preparation. In the new process it is reduced to 5%!

9

Useful Notions

• ADM=Analytical Data Mart• ABT=Analytical Base table• Input variable=variables, which explain the

target.• Sandbox= experimental input variables• Target variable=if a customer buy specific

product in a timeperiod• Metadata driven (macro)= add a new product

is just filling a excelsheet.

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Defining Rules

• Identify the target.• Identify Target group

• Nse=New sale existing customers• Nsp=New sale prospect• Nss=New sale suspect• Uds=Up/down sale• Ups=Up sale

• Upsale one step up.• Extra filters, 18 years and older, etc.

10

Mbb_<…>

Mpr_<…>

Mpp_<…>

CuCu_<…>

ABT

Fix_<…>

Dsl_<…>

Cu_<...> Cu – level for NSP, NSE, NSS Models

Mbb_<…> Cu_<...>

Targ

TargMBB

MPP

MPR

DSL

FIX

Mpp_<…> Cu_<...>Targ

Mpr_<…> Cu_<...>Targ

Dsl_<…> Cu_<...>Targ

Fix_<…> Cu_<...>Targ

Abt_Master_Cu

Abt_Master_Mbb

Abt_Master_Mpp

Abt_Master_Mpr

Abt_Master_Dsl

Abt_Master_Fix

Cu_<…>Sandbox Cu_<...>

ADM

Sub level – for UDS, UPS• The target variables

of potential modelsare calculated everymonth(abtmaster.sas).

• To select the right abtfor a specific model is easy (abtmodelling.sas).

• Last month• All months• Selection of a

month

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Data Exploration in SAS Visual Analytics to get a first feeling

11

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Transform & Select the Right Input Variables withMaximum Predictive Power• Numeric encoding for high-cardinality nominal variables such as zip code.• Normalizing, binning, log transformation for interval variables.• Transformations based on missingness patterns.• Dimension reduction techniques such as autoencoders, principal component analysis (PCA), t-Distributed

Stochastic Neighbor Embedding (t-SNE), and singular value decomposition (SVD).

12

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Analytical Modeling

• Many different algoritms in Sas Enterprise Miner available• Decision tree• Regression• Neural Network• Gradien Boosting• Random Forest

• Model comparison node for comparing whichmodel is the best.

13

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Validate models

• Is the initial model better than the champion model (old model)• Validation and approval of the champion model

14

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Deploy (scoring) a model is easy!

• Models are available for many Sas application• Sas CI Studio• Sas Enterprise Guide• Sas DI studio• Sas Model manager• Sas RTDM• Sas Esp (A-store)

15

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

Easy to monitor the model, so we can react fast.

Monitoring• Variable distribution• Lift• Gini (ROC)• Kolmogorov-Smirnov (KS)

Threshold• AUC decay• Lift decay

16

Results/decisions• Recalibrate a model• Retire a model (new)

Identity business problem

Data preparati

on

Data explorati

on

Transform &

selectAnalytic

almodelin

g

Validatemodels

Deploymodels

Evaluate/monitor results

17

Next challenge #2: Fully AI enabled customer journeyoptimization

Telenor Research:

Developing deepreinforcement modelto optimize customerjourney, based on all the interactions of the

customer.

Thank youHans de Wit, Telenor Mobile Norway, +47 48 29 1399