Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge

7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge

1/24

Faraday faraday.io


2/24

Faraday

Tackling the customer acquisition challenge2

Introduction

TH E CUSTOMER ACQUISITION PR OB LEM

Chances are that if youre a company selling a considered consumer purchase

(>$1,000 ticket size), you have a problem: acquiring customers is too expen-

sive.

At each step of the big-ticket customer acquisition process there are costs, in-

cluding marketing to prospects, engaging with leads, and potentially, depend-

ing on your industry, on-site assessments. This problem is exacerbated by

that fact that customer engagement around considered purchases is notori-

ously difficult. For example:


3/24

FARADAY WHIT EPAPER SERIES 3

faraday.io

The average U.S. consumer spends just seven minutes a year interact-

ing with their electricity provider.1

41% of individuals during the open enrollment period for health care

in 2013 spent 15 minutes or less researching their options. 2

OUR SOLUTION

To begin to address these challenges, Faraday has developed

a data-driven marketing platform optimized for companies

selling considered purchases. The platform includes a machine

learning-driven prediction engine and automated marketing

instrumentation, empowering marketers to identify promising

prospects, target them, and track outcomes.

To test the platform, Faraday teamed up with one of its clients

in Massachusetts. The goal was to use the Faraday platform

to help the client reduce acquisition costs by more effectively

identifying and engaging with the individuals most likely to

accept. This paper presents our approach and the results.

1 Accenture: Actionable Insights for the New Energy Consumer

2 USA Today: Many Spend 15 Minutes or Less Picking Health Insurance

MACHINE LEARNING is a classof techniques that allow computers

to learn information that was not e

plicitly provided. Machine learning

is widely used in predictive data

analysisfamiliar examples includ

the Netflix and Amazon recommen

dation engines. Faradays platform

uses supervised classification algo

rithms to identify patterns in samp

datasets that can be generalized to

make predictions for new data.
http://www.accenture.com/us-en/Pages/insight-actionable-new-energy-consumer.aspxhttp://www.usatoday.com/story/money/personalfinance/2014/09/04/health-insurance-plans-costs/15032405/http://www.usatoday.com/story/money/personalfinance/2014/09/04/health-insurance-plans-costs/15032405/http://www.accenture.com/us-en/Pages/insight-actionable-new-energy-consumer.aspx


4/24

Faraday


The project

BACKGROUND

The project was designed to test whether Faradays data-driven marketing

platform would be an asset for driving down customer acquisition costs for

our client. The project used property data, consumer data, and historical

CRM (marketing and customer data) for 15 towns in the Greater Boston met-

ropolitan area to develop a model that predicts the likelihood that a household

would invest in our clients product. We tested the model with a direct mail

campaign sent to a mix of high-scoring and randomly selected households,

none of which were existing customers. We instrumented the campaign to

allow automatic real-time response tracking, making it easy to evaluate per-

formance and incorporate the findings into an improved model for follow-on

campaigns.


5/24


faraday.io

ELIGIB ILITY

To match the eligibility requirements for our clients product we excluded

non-residential buildings and multi-family buildings with more than four

units. We also excluded buildings built after 1980 because past market seg-

mentation strategies left us with insufficient data on newer homes. There

were 93,823 housing units that satisfied these criteria, representing just

under half of the 15 towns combined housing stock.

Table 1.Number of eligible homes in each project town3

TOW N H OU SIN G UN ITS E LIGIBL E U NITS PE RC EN T EL IGIBLE

Acushnet 3,985 2,312 58%

Ashland 6,578 2,324 35%

Cambridge 48,278 16,337 34%

Carver 4,506 1,685 37%

Dedham 10,098 6,895 68%

Fairhaven 7,446 4,978 67%

Framingham 27,535 14,955 54%

Holliston 5,051 3,552 70%

3 Housing units data from U.S. Census Bureau American Community Survey (2012)


6/24

Faraday


TOW N HO USIN G U NITS E L IG IBL E UN ITS PERC E NT E LIGIBL E

Hopkinton 5,038 1,935 38%

Maynard 4,425 2,530 57%

Needham 10,822 7,402 68%

Plymouth 25,288 10,389 41%

Sherborn 1,452 1,102 76%

Somerville 32,471 13,773 42%

Westwood 5,312 3,654 69%

DEMOGR APH ICS

The 15 towns range in size from Sherborn, with a population of about 4,000,

to Cambridge, with over 100,000 residents, and have a combined population

of approximately 472,000 living in almost 200,000 housing units.


7/24


faraday.io

Figure 1.Map of project towns

At one end of the socioeconomic spectrum Sherborn has a median household

income of $150,000 and 83% of the residents over age 25 have a bachelors de-

gree. At the other end Fairhaven has a median household income of $60,000

and only 23% of its residents over age 25 have a bachelors degree.

The towns housing stock shows similar variation. In Somerville 90% of hous-

ing units were built before 1980, just 15% are single-family homes, and two


8/24

Faraday


thirds of residents rent. In contrast over half of Hopkintons housing stock

was built since 1980, 90% is single-family homes, and only 10% of residents

are renters. Carver is a unique case as the only town with a significant num-

ber of mobile homes, with these accounting for almost a quarter of the towns

housing units.

Table 2.Size and socioeconomic characteristics of project towns4

TOWN POPULATION MEDIAN INCOME POVERTY BACHELOR'S DEGREE

Acushnet 10,302 $65,222 5% 22%

Ashland 16,587 $95,296 4% 55%

Cambridge 105,026 $72,225 14% 74%

Carver 11,497 $67,963 7% 22%

Dedham 24,716 $82,193 5% 45%

Fairhaven 15,893 $59,933 10% 23%

Framingham 68,689 $68,906 10% 45%

Holliston 13,668 $107,192 4% 60%

Hopkinton 14,982 $127,821 2% 69%

Maynard 10,151 $79,441 4% 49%

Needham 29,005 $125,170 4% 73%

Plymouth 56,574 $77,228 6% 33%

4 ibid


9/24


faraday.io

TOWN POPULATION MEDIAN INCOME POVERTY BACHELOR'S DEGREE

Sherborn 4,128 $151,944 3% 83%

Somerville 75,974 $64,603 16% 53%

Westwood 14,616 $130,824 3% 68%

Table 3.Housing unit characteristics of project towns5

TOWN MEDIAN VALUE S INGLE-FAMILY PRE-1980 RENTERS

Acushnet $299,200 81% 64% 16%

Ashland $353,900 76% 50% 19%

Cambridge $545,800 16% 80% 64%

Carver $269,100 74% 51% 8%

Dedham $374,700 72% 80% 27%

Fairhaven $272,500 76% 82% 27%

Framingham $349,800 54% 85% 44%

Holliston $386,000 85% 78% 13%

Hopkinton $510,000 93% 44% 10%

Maynard $333,000 69% 80% 35%

Needham $651,300 81% 77% 17%

Plymouth $336,200 80% 62% 20%

5 ibid


10/24

Faraday


TOWN MEDIAN VALUE SINGLE-FAMILY PRE-1980 RENTERS

Sherborn $721,100 94% 76% 10%

Somerville $441,300 15% 90% 67%

Westwood $620,400 84% 76% 12%


11/24


faraday.io

Methodology

DATA SOURCES

MassGIS The base input to our model was publicly available

town assessor and geospatial data from MassGIS.6This mainly

provided building variables such as location, area, year built,

type, number of units, assessed value, and last sale date.

U.S. Census We also added block-level data from the U.S.

Census such as median household income and educational

achievement.

6 MassGIS

While this project made use of pub

licly-available datasets like MassG

and the U.S. Census, the Faraday

platform is built on a nationwide

database which comprises multi-

ple commercially licensed house-

hold-level datasets.
http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/


12/24

Faraday


Consumer data We enhanced the model with consumer data purchased

from a national data aggregator. This included a large number of variables

such as owner/renter, occupation, and environmental inclination.

Clients CRM data We used historical Cus-

tomer Relationship Management (CRM) data,

which included previous marketing efforts that

previously accepted or rejected an offer for our

clients product. Homes that accepted an offer

were easy to identify, but many homes simplydid not respond to prior marketing efforts. Fur-

ther investigation revealed that unresponsive

homes with three or more outreach attempts

were highly unlikely to go on to our clients offer, so we classified these homes

as did not invest. Unresponsive homes with fewer than three outreach at-

tempts were excluded from the analysis.

PR EDICTIVE MODEL

We used an iterative process to create a predictive model across all 15 pilot

towns. Working with only the homes classified as invested or did not in-

vest we created a large number of feature sets by randomly selecting subsets

of the available variables. For each of these feature sets:

INVESTMENT is a generic term for

a familys purchase or adoption of a

good or service.


13/24


faraday.io

1. We randomly split the homes into a training set containing 70% of the

homes and a test set containing the remaining 30%.

2. We used the training set to train a random decision forest ensemble

to predict whether each home did or did not accept an air sealing or

insulation offer.

3. We evaluated the model by using it to predict the out-

come for the homes in the test set and comparing the

predictions to the actual outcomes.

We performed this split, train, and evaluate process 10 times

for each feature set and averaged the results. We chose the

best-performing feature set and used it to train a random forest

model on the combined training and test sets. The resulting

model takes the feature sets variables as inputs and outputs a

score from 0-100 representing the likelihood that a household

will invest in our clients product.

MAILING LIST

We used the model to score all eligible homes in the 15 pilot

towns. We then created a mailing list of 28,095 homes that had

never purchased our clients product. Half were a randomly

RANDOM DECISION FORESTSare a technique in machine learning

a branch of artificial intelligence.

They involve splitting datasets

with known outcomes at randomly

chosen points in randomly chosen

attribute dimensionsrepeatedly.

Sometimes, these splits increase

the models ability to discern good

outcomes from bad; these splits

are kept, others are discarded. The

resulting forest of decision trees

can subsequently vote on a new

candidates likely outcome based o

where its attributes fall.


14/24

Faraday


chosen control group and half were the highest-scoring homes not already

allocated to the control group.

INSTRUMENTATION

Faraday provided a unique phone number to be printed on each mailer. Calls

to the number are automatically linked to the mailer recipient in a databaseand are viewable in real-time through Faradays campaign tracking tool.


15/24


faraday.io

Results

SUMMAR Y METR ICS

We used four metrics to evaluate feature sets for the final model:

Precision The fraction of homes the model predicted would invest that did

invest. The higher the precision, the more likely it is that homes the model

predicts will invest do invest.

Recall The fraction of investors that the model predicted would invest. High-

er recall means fewer missed opportunities; fewer investors that the model

predicted would reject.

Accuracy The fraction of the models predictions that were correct. Higher


16/24

Faraday


accuracy means the model made fewer mistakes in predicting both investors

and rejectors.

F1 score The harmonic mean of precision and recall. A higher F1 score

means the model is better at picking out investors. The key difference be-

tween F1 score and accuracy is that F1 score is more influenced by correct

investor predictions. Since we care more about identifying investors than

rejectors the F1 score is more relevant than accuracy.

Table 4 lists the metrics for the best-performing feature set. As describedin the methodology section these values are averaged across 10 models and

were calculated by making predictions for data that the models had never

seen. Thus they give an indication of how the model should perform in future

marketing efforts.

Table 4.Summary metrics for top feature set

PRECISION RECALL ACCURACY F1 SCORE

80% 31% 72% 45%


17/24


faraday.io

TOP PREDICTIVE VARIABLES

Faradays model analyzed more than 130 variables describing housing unit

characteristics and consumer behavior. The two most predictive variables

were the household heads occupation and influence rating on social net-

works. Other top variables included the number of children, home value, and

a buildings siding type.

While these variables proved most predictive for the pilot project, Faradayhas found that predictive variables for one geographic area or product do

not necessarily perform well in other areas or with different products. This

means a universal model will be less predictive than a model tuned to the

unique demographic, financial, and property characteristics of a particular

market.

ROC CHART

A receiver operating characteristic or ROC chart is useful for deciding how to

use a models predictions. It shows how recall and false positive rate change

as the discrimination threshold varies. False positive rate is the fraction of

rejectors that the model predicted would invest. The discrimination thresh-

old is a cutoff score below which we no longer predict that a household will

invest.


18/24

Faraday


Figure 2 is based on the final models predictions for all investors and rejec-

tors. The black curve shows the result of using different cutoff scores while

the grey diagonal shows the result of randomly guessing whether each home

will invest. The black curve labels indicate the cutoff score. The greater the

distance between the curve and diagonal the bigger the performance increase

from using the model.

Above a cutoff score of 66 the model yields a huge improvement: over 60% of

the investors are included with less than 1% of the rejectors. Below this point

the model adds rejectors at a greater relative rate than investors.

Figure 2.ROC chart


19/24


faraday.io

GAINS

A gains chart illustrates the extent to which a smaller marketing effort

informed by a predictive model could have resulted in some or all of the

investments. In Figure 3 the black curve shows the result of targeting homes

in descending order by score. The grey diagonal shows the result of random

targeting. The black curve labels indicate the cutoff score. The greater the

distance between the curve and diagonal the bigger the performance increase

from using the model.

A list of the top 25% of homes by score contains over 60% of the investors and

a list of the top 50% of homes contains over 80% of the investors. This sug-

gests that future marketing based on the model could cut costs in half if a 20%

reduction in investments is acceptable.


20/24

Faraday


Figure 3. Gains chart

MAILING LIST PER F OR MANCE

The ultimate test of the models performance will be the relative rates at

which the model-selected and randomly selected mailer recipients invest

in our clients product. Four months after receiving the offer, recipients

showed a statistically significant 25% higher response rate than the control

group. The percentage of customers that signed contracts to complete further


21/24


faraday.io

weatherization improvements was 33% higher for the top scoring group with-

in 60 days of receiving a letter.

Figure 4.Mailing list response rate


22/24

Faraday


Conclusion

Both the model summary metrics and the initial results of the test mailingsuggest Faradays data-driven customer targeting increases customer acqui-

sition rate. The model ranked over 80% of past investors in the top 50% of

all households by likelihood to invest and the pilot mailing shows an initial

33% increase in conversion rate. The models performance may improve for

follow-on campaigns as it leverages data from previous campaigns to refine

its predictions.

The increase in response rate and conversion rate means that in the pilot

area our client is on track to save almost $13,000 on mailers and $311,000 on

reduced sales costs compared to a traditional campaign. If we assume a more

conservative 20% lift for a statewide campaign, our client could mail 17%

fewer homes resulting in savings of $540,000 on mailers and $13 million in

sales costs.7

7 Assuming an average of 1.5 mailers per home, $1 per mailer, and $400 in sales costs.


23/24


faraday.io


24/24

Matt Pell

[email protected]

(802) 458-0441 x341

Scott Pellegrini

[email protected]

(802) 458-0441 x340

Faraday

Documents

Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge