Ona

All About the User Experience:

Data Science at MailChimp

John Foreman

September 2013

OR guy

Former consultant

I heart large, slow organizations

Most companies are

not in the business of

analytics

Improve the product

Improve the user experience

Improve the product

Improve the user experience

Internal External

Insights

Capabilities

Customers

Internal External

20%

80%

Insights

Capabilities

Internal External

Insights

Capabilities

Internal External

Insights

Capabilities

“Send email, all

the time, as

much as you

can…Obama.”

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

0 5 10 15

Ca

mp

aig

n C

lic

k R

ate

Sends per Month

User X: Sends per month vs

campaign click rate

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

3.5%

4.0%

4.5%

5.0%

0 10 20 30

Ca

mp

aig

n C

lic

k R

ate

Sends per Month

User Y: Sends per month vs

campaign click rate

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

3.5%

4.0%

4.5%

5.0%

0 10 20 30

Ca

mp

aig

n C

lic

k R

ate

Sends per Month


campaign click rate

0

50

100

150

200

250

0 10 20 30

To

tal m

on

thly

clic

ks

Sends per Month


total monthly clicks (list size 1000)

Internal External

Insights

Capabilities

Compliance

Sends campaign w/

high bounce rate

Sends campaign w/

high unsub rate

Sends campaign w/

high abuse rate

Warning (Still

Enabled)

Disable/Review

Replies

Ignores/Waits/A

cknowledges

Disabled

Enabled

Review

(can still send

at current level)

User Action Consequence User Action Final State

Replies

Ignores/Waits

Disabled

Enabled

High bounce rate

High unsub rate

High abuse rate

Warning (Still

Enabled)

Disable/Review

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

Warning

Disable

Review

(can still send

at current

level)

Direct complaint

Hardcoded spam rule

Causes IP blacklisting

Hits URL blacklist

Multiple accounts

No unsub link

TOS violation

Large List Upload

Permission reminder

Tries to buy $X+

Requests HV approval

33%

17%

2%

1%

3%

19%

3%

1%

3%

2%

14%

2%

Sends campaign w/

high bounce rate

Sends campaign w/

high unsub rate

Sends campaign w/

high abuse rate

Warning (Still

Enabled)

Disable/Review

Replies

Ignores/Waits/A

cknowledges

Disabled

Enabled

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

High bounce rate

High unsub rate

High abuse rate

Warning (Still

Enabled)

Disable/Review

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

Warning

Disable

Review

(can still send

at current

level)

33%

17%

2%


Warning (Still

Enabled)

Disable/Review

Replies

Ignores/Waits/A

cknowledges

Disabled

Enabled

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

Warning (Still

Enabled)

Disable/Review

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

Warning

Disable

Review

(can still send

at current

level)

Large List Upload

Tries to buy $X+


3%

2%

14%


“Account Review

takes too much

time!

My needs is

gone!”

Stop reviewing

accounts every

time they make a

change. We paid

you for

a service…your

service is too

restrictive…

“Bullshit review process...I'm sending out high Press Releases about this

time wasting process...Google me and you would know I'm a serious

marketer with influence. Ill definitely make sure everyone I know in this

industry knows what a pain in the ass this was.”

Permission

Hygiene

Permission

Hygiene

Purchased

Hard Bounce

Final modeling

setInter-

mediate Modeling

Data Collection

Category

UserAcct

metadataAcct

Metadata

List

Address history Bounce

ModelList scores

Campaign

Campaign history

Prev. camp. perf.

ContentContent Model

Content scores

Predictions

Fin

al M

odelPublic Lists

Design to eliminate

risk/complexity

Function Tool

Slow Storage Sharded PostgreSQL

Fast Storage Redis

Modeling Language R

Machine Learning Model Random Forest

API + Background Processes Python

Sends campaign w/

high bounce rate

Sends campaign w/

high unsub rate

Sends campaign w/

high abuse rate

Warning (Still

Enabled)

Disable/Review

Replies

Ignores/Waits/A

cknowledges

Disabled

Enabled

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

High bounce rate

High unsub rate

High abuse rate

Warning (Still

Enabled)

Disable/Review

Review

(can still send

at current level)

Replies

Ignores/Waits

Disabled

Enabled

Warning

Disable

Review

(can still send

at current

level)

Large List Upload

Tries to buy $X+


33%

17%

2%

3%

2%

14%

70+% tickets automatable

Internal External

Insights

Capabilities

If I am to speak 10 minutes, I need a week for prep;

If 15 minutes, three days;

If half an hour, two days;

If an hour, I am ready now.

-Woodrow Wilson

Internal External

Insights

Capabilities

Customers

Communicating

Technical

HARD

A product is only as powerful

as it is easy to use

Example: Discovered Segments

Example: Discovered Segments

Make Data More Valuable

Inside

the Product than

Outside

Align Yourself & Serve


No Special Treatment



1) Goal 2) Toys



1) Goal 2) Toys

Avoid Complexity

Thanks!@John4man

Technology

Ona