More Data Mining Success Stories for Marketing and Related FieldsWolfgang JankRH Smith School of BusinessUniversity of Maryland

What is “Data Mining”?

What is Data Mining?

Many Definitions Non-trivial extraction of implicit, previously

unknown and potentially useful information from data

Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns

Related Fields

Statistics

MachineLearning

Databases

Visualization

Data Mining and Knowledge Discovery

Why Mine Data?

Because there are Data Floods….

Lots of data is being automatically collected and warehoused Web data, e-commerce Scanner data at department/

grocery stores Bank/Credit Card/Insurance

transactions

Computers have become cheaper and more powerful

Competitive Pressure is Strong Provide better, customized services for an edge

Why Mine Data?

Big Data Examples

Europe's Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation session storage and analysis a big problem

AT&T handles billions of calls per dayso much data, it cannot be all stored -- analysis

has to be done “on the fly”, on streaming data

Data Growth

In 2 years, the size of the largest database TRIPLED!

Data Mining is particularly promising Online

Why?Because every “click” leaves a digital footprintWe can use these footprints to better

understand our customers… Coupons, ads, discount, dynamic pricing, …

…or guard them against predators Fraud detection, account protection, spam, junk

mail, viruses, …

Blog Pulse

Measures what the world (= the internet) is thinking Measured in

terms of the blogging activity

The “Obama Buzz” started here!

The Republican Convention & Sarah Palin

Google Trends

Measures what the world is looking for Measured in

terms of search words

The world’s interest in “Lehman Brothers” and “AIG”

Google Flu Trends Detects outbreaks of

flu early and only based on search terms More accurate and

faster than CDC Read more at

http://www.google.org/flutrends/

Data Mining Success Stories

The Netflix Recommendation Engine Netflix uses data mining to

make recommendations to its users Based on past user behavior Based on movie similarities

Helps cross-selling of products

Improves the search experience for users

However, developing good recommendation engines is not easy; therefore, Netflix has initiated the Netflix Challenge

The Netflix Challenge

Netflix offers $1 million for the person/team that can improve their current data mining method by 10% (i.e. classification accuracy) http://www.netflixprize.com/ Incremental progress prizes of $50,000 every year AT&T team has won progress prize in 2007

“The Netflix Prize seeks to substantially improve the accuracy of predictions about how much someone is going to love a movie based on their movie preferences”

Amazon’s Recommendation Engine

Every time we buy a book on Amazon, we receive recommendations about similar books

How are they doing this?

The answer: massive data mining

Google’s Search Algorithm

Google continuously collects data about web pages using web spiders

It transforms this massive data into search information using the famous “page-rank” algorithm

AT&T’s Fraud Detection

Name Elizabeth Harmon

Address APT 1045

4301 ST JOHN RD

SCOTTSDALE, AZ

Balance $149.00

Disconnected 2/19/04 (nonpayment)

Name Elizabeth Harmon

Address 180 N 40TH PL

APT 40

PHOENIX, AZ

Balance $72.00

Connected 1/31/04

Fraudulent account: terminated!

Should this new account be allowed?

In the AT&T telephone network, every day old nodes drop out (terminated accounts) and new nodes pop up (new accounts)

AT&T’s Fraud Detection

AT&T uses massive graph mining to detect fraud in their telephone network data

Mining Accounting Fraud at PricewaterhouseCoopers

PwC uses data mining for the automatic analysis of company general ledgers to detect accounting fraud

Helps conform with Sarbanes-Oxley Act Improves efficiency Improves accuracy

Sales Lead Identification at IBM

IBM uses predictive modeling to estimate opportunities for cross-selling to existing customers, selling of existing services to new customersUses analytic tools to estimate

A potential customer’s wallet size A potential customer’s probability of purchasing

a service

Data Mining at IBM

New Rational sales

Historical System p sales

Historical total Software sales

State is CA

Sector is IT

Company is HQ

Historical Lotus sales

Historical System x sales

IBM RelationshipFirmographics

Historical System z sales

zata3: Data-Driven Decisions in Election Campaigns

zata3 is an election campaign consulting company

They recently decided to add data mining technology to their services

zata3: Lot’s of data on voters and past voting behavior

PARTY_CODE Gender Education Children Home_Owner Income Times DonatedA 0 3 1 4 3 0R 0 0 4 2 0A 1 0 3 1D 1 0 0 0D 1 5 0 2 6 0R 1 0 4 3 0R 1 0 4 1 0R 1 0 0 0D 1 7 1 2 4 0D 1 0 0 0

General(00-03) Presidential Primary VH General VH Presidential VH Primary G04 VotedY 1 0 0 0YYYY R 4 1 0 1YY 2 0 0 1

0 0 0 1YYYY DD 4 2 2 1YYY RR 3 2 1 1YYYY RR 4 2 1 1

0 0 0 1YYYY DD 4 2 3 1

0 0 0 1

Goal: to predict who will vote in the next election

Idea: better targeted spending of election campaign resources

zata3: Huge savings with data mining Zata3 anticipates savings of over 30%

using data mining models

Traditional Total Cost Voted Cost Per Vote74,522.50$ 14664 5.08$

With Data Mining Total Cost Voted Cost Per Vote52,806.64$ 15626 3.38$

Savings Total Cost Votes Cost Per Vote % Savings21,715.86$ 962 1.70$ 34%

Analysis

Data Mining and Mass e-Customization

Customization for Online Services

Opportunities: Combination of countless

features for highly individualized solutions

“A single personalized solution for every customer”

Challenges: How does the customer

understand what’s right for them?

Moving from consultative selling to self-consultative buying

Ex.: Freddie Mac Mortgage Services Freddie Mac mass

customizes mortgage products Combines hundreds of

different loan characteristics

Challenge: How does the customer find the loan that’s right for them?

Ex.: Mass Customization at eBay

eBay offers any possible product & service in “garage-type” sales However, it does not assist the customer much in

finding the right product/service.

Ex.: Books on Amazon

Amazon.com offers books for every taste But: How can we find the book that’s right for us?

Managing Mass Customization at Amazon How does Amazon assure that customers

find what they are looking for?Answer: by making (automated)

recommendations

Managing Mass Customization From Expert Salesperson to Expert System:

How can we assure that our customers get what they are looking for?

Pre-Internet customization: Expert Salesperson

Experienced with product, process

Consultative selling Salesperson provides

expertise, identifies needs, defines configuration

Early/current-Internet customization: Expert Customer

Experiences with product Revelation, Transaction buying Customer provides expertise,

knows needs, defines configuration

Future Internet Customization: Non-Expert Customer

Inexperienced with product, process

Self-consultative buying System provides expertise,

identifies needs, defines configuration

Providing the non-expert customer with decision support

Moving from Expert to Non-Expert Buyers: Computerization

Assisted service Telephone, email,

instant messaging Drawback: requires

human interaction, only limited scalability

Self service Search, user ratings, forums,

blogs, expert recommendations

Drawback: does not help the customer that is unsure about their needs

Automated service Expert systems for the non-expert Replaces the salesperson Translates customer characteristics and usage requirements into

recommended product configurations Consists of rule-based systems and data mining algorithms Advantage: fully automatic, scalable, updatable

Ex.: Automated-Service at AmEx Offers online tool that, based on desired

features, recommends best card Compensates only for lack of product knowledge,

but assumes customer knows why they need the product.

Ex.: Blockbuster’s Recommendation System

Blockbuster recommends similar movies based on movie features and user behavior “If you liked Indiana

Jones, then you will also like Tomb Raider”

Key Component for Automated Service Systems: Data Mining

Collect and mine customer information in order to, e.g., Segment the market

Understand customers’ different needs, expertise, profitability E.g. Dell distinguishes between the segments “Home”, “Small

Business”, “Medium/Large Business”, “Public Sector” Analyze behaviors and events

Understand when customer has needs and the events that lead to them

E.g. path tracking, click stream analysis Optimize prizing

Bundling, price discrimination E.g. Amazon’s price testing; Zilliant’s data-driven pricing software

Key requirement: understand customer data

Dangers of Data Mining

Dangers of Data Mining

The danger of using data mining software/technology as a “black box”Data does not mine itself!We still need the domain knowledge and

expertise of the user; otherwise outcomes may be meaningless

Data qualityJunk-in, junk-out

What Data Mining Isn’t

Data Mining Isn’t… …smarter than you

Example from DeVeaux: A new backpack inkjet printer is showing higher

than expected warranty claims A neural networks analysis shows that Zip code is

the most important predictor

Data Mining Isn’t… …always about algorithms

Sometimes collecting an plotting the right data is enough

Blogpulse

More Data Mining Resources

Repository:http://www.kdnuggets.com/http://www.the-data-mine.com/

Tutorialshttp://www.autonlab.org/tutorials/

SoftwareSAS Enterprise Miner, SPSS Clementine,

Orange, Weka, Rattle, R, …