Predictive Analytics over On-line and Social Network Data · 2010-06-08 · Predictive Analytics over On-line and Social Network Data Usama Fayyad, Ph.D. CEO, Open Insights ... Spring

Open Insights

Predictive Analytics overOn-line and Social Network Data

Usama Fayyad, Ph.D.CEO, Open Insights

www.open-insights.com

Copyright 2009 by U.M. Fayyad, All rights reserved0


Open Insights

Some of this material extracted from prior talk: The Evolution of Search Technology

and the Role of Social Networking in Marketing

Research

Usama Fayyad, Ph.D.CEO, Open Insights

1

Acknowledgement

Some of the work presented in This talk was done in

former position:

Chief Data Officer & Executive VP

Yahoo! Inc.

Copyright 2009 by U.M. Fayyad, All rights reserved www.open-insights.com

Background and Lessons

• Grad student forever… at University of Michigan, Ann Arbor

– after 5 degrees, Lesson: real world can be more exciting

• Jet Propulsion Lab, California Institute of Technology (1989-95)

– Data mining can help the highly specialized tremendously

• Microsoft (1995-2000)

– The real problem is not data mining technology, but operations

• digiMine Inc. (2000-2003) – [now Audience Science]

– The problem is explaining the strategic value of Data to exec team and connect with business strategy

• DMX Group (2003-2004+)

– Data strategy in business intelligence and delivering on the deeper value

• Yahoo! (2004 – 2008)

– The Internet is a much deeper social and commercial transformation than I ever imagined

• Open Insights (October 2008 - )

– Back to Data Strategy and Data Mining

2


Overview• Yahoo! and On-Line Marketing Primer

• From Search/Advertising to Marketing:

– The Evolution of Search and search marketing

– More focus on Search Marketing

• Case-study: Impulse (Search-after)

• Case Study: Behavioral Targeting

• Case Study: Retargeting

• Case Study: Choicestream ad personalization

• The Evolution of “Social Media” (Introduction)

• Wisdom of the Crowds

• Case Study: Social Targeting at Yahoo!

• Case Study: Social Targeting using Twitter (VH1 example)

• What Does this mean for Search?

– Where does Search evolve to?

• Concluding Thoughts

3


506.3

602.4

702.4

787.5

868.3

974.3

1,076.7

0.0

200.0

400.0

600.0

800.0

1,000.0

1,200.0

2001 2002 2003 2004 2005 2006 2007

Worldwide Total Japan

Rest of World Western Europe

Asia/Pacific United States

Globally, Internet Users Outnumber 1 Billion by 2007

Source: IDC, December 2003.

Internet Users in Millions:

4


Yahoo! is the #1 Destination on the Web

73% of the U.S. Internet population uses Yahoo!

– Over 500 million users per month globally!

• Global network of content, commerce, media, search

and access products

• 100+ properties including mail, TV, news, shopping,

finance, autos, travel, games, movies, health, etc.

• 25 terabytes of data collected each day… and

growing

• Representing thousands of cataloged consumer behaviors

More people visited

Yahoo! in the past

month than:

• Use coupons

• Vote

• Recycle

• Exercise regularly

• Have children

living at home

• Wear sunscreen

regularly

Sources: Mediamark Research, Spring 2004 and comScore Media Metrix, February 2005.

Data is used to develop content, consumer, category and campaign insights for our

key content partners and large advertisers

5


Yahoo! Data – A league of its own…

Terrabytes of Warehoused Data

25 49 94 100500

1,000

5,000

Amaz

on

Kore

a

Tele

com

AT&T

Y! L

iveS

tor

Y! P

anam

a

War

ehou

se

Wal

mar

t

Y! M

ain

war

ehou

se

GRAND CHALLENGE PROBLEMS OF DATA PROCESSING

TRAVEL, CREDIT CARD PROCESSING, STOCK EXCHANGE, RETAIL, INTERNET

Y! Data Challenge Exceeds others by 2 orders of magnitude

Millions of Events Processed Per Day

50 120 225

2,000

14,000

SABRE VISA NYSE YSM Y! Global

6


What we needed at Yahoo!

• Data Strategy

• Prioritization conducted based on business need, not IT

– Drove new innovations in DB

– Data segmentation management

– Specialized stores

– File systems for grid computing (Hadoop)

• Had to resort to own home-grown data stores for largest applications

7


Advertising: Brand and DR

Knowledge of users & their behavior throughout the purchase funnel can

grow brand & direct response revenue

Awareness

Purchase

Consideration

> $220B Brand Advertising Market

> $250B Direct Marketing Market

Most time & activity is in consideration & engagement, but there are limited metrics & reach strategies

8

Open Insights

A Tale of Two Search Engines

Research 9



Algorithmic results

=Audience

Advertisements

=Monetization

-$ +$

10


Algorithmic vs. Ad Search

• Analogous to classical separation of editorial vs commercial content

• Technical underpinnings:

– Some commonalities (IR, ML)

– Many differences (incentives, spam, mechanism design)

• Will cover both algo and ads

11

Open Insights

Search advertising

12



Search query

Ad

13


A Question for the audience

• Do you think an “average” user, knows the difference between sponsored search links and algorithmic search results

• Do you think an “average” user knows there are sponsored links on the page?

• Do you think a user knows where a sponsored link would navigate to upon a click?

14


Ads go

in slots

like

these

15


Higher

slots

get

more

clicks

16


Economic ordering

• Yahoo!/Overture started with Bid Ordering of search adsThe only business model at the time

• Google introduced revenue ordering to avoid the Overture patent

– Resulted in 4x – 6x revenue per search!

• Bid and revenue ordering: two forms of ordering by an econscore

– Auction mechanism matters (Vickrey Auction)

– Is not the right generalization to Search Ads (Edelman, Ostrovsky, Schwarz 2006)

• Does revenue ordering maximize revenue?

• No – advertisers react to ordering scheme, by changing their bid behavior!

• Lahaie+Pennock ACM EC 2007

– Family of schemes bridging Bid and Revenue ordering

– Game-theoretic analysis

17


A new convergence

• Monetization and economic value an intrinsic part of system design

– Not an afterthought

– Mistakes are costly!

• Computing meets humanities like never before –sociology, economics, anthropology …

• We call it Computational Advertising

18

Open Insights

Why is search-related advertising so powerful?

A question for the Audience:

19


Moving Customers up the Funnel

Impulse Banners– Target users based on their activity – both search and

property -- within the NEXT HOUR

• Behavioral Categories – Apparel, Computers, Home Appliances – all the same categories that you can use for regular behavioral targeting!

Awareness

Purchase

Consideration

20


Impulse Banner Example

All within

1 HOUR2

Sees that “Credit

Card” falls under

the category:

“Finance/Credit

and Credit

Services”

25% - 261% higher CTR than RON

1

User

searches on

the word

“Credit

Card”

3

serves

“Finance/Credit and

Credit Services”

banners to User

anywhere on the

Yahoo! network

within 1 HOUR

21


Way Impulse Works

• Searches are not at all associated or tracked through personally identifiable information

• No long-term memory of search terms, all stored on client cookie.

• We generalize the category is targeting is at generic category: e.g. Financial Services, not “credit card”

• All targeting done in anonymous mode

22


BT2.0 vs. Impulse (test data)

+475.8%

CTR Lift

+291.5%

Conv Lift

*Note: these are straight averages rather than weighted averages. Therefore, might not be the best representation of the true overall CTR lift.

•Shoppers Testing (18 tests across 10

verticals)

CONCLUSION:

BT2.0 can utilize

Search+Behavior Data to

deliver higher effectiveness,

and much larger inventory

than targeting by Search

queries only!

CTR Lift

Over

Impulse

Reach lift

Over

Impulse

Autos 171% 254%

CPG 147% 308%

Entertainment 156% 559%

Finance 264% 404%

Health & Pharma 70% 588%

Life Stages 75% 493%

Retail 65% 424%

SMB & B2B 116% 984%

Sports 222% 429%

Tech 129% 867%

Travel 59% 246%

Average 144% 652%

23


Behavioural Targeting (BT)

Search

Ad Clicks

Content

Search Clicks

BT

24


What‟s „Behavioural Targeting‟?

• Targeting your ads toconsumers whoserecent behaviors online indicate that your product category is relevant to them...

25


Moving Down the Funnel

• New generation marketing solutions to take brand advertisers down the marketing funnel

Awareness

Purchase

Consideration

26


AUTO

S / SE

DAN

Beha

vioura

l Inter

est

Sedan

What goes into user behavior models?

• Relevant behaviors…

• Pageviews

• Ads clicked

• Search queries

• Search clicks

27


How it works | Network + Interests + Modelling

Analyze predictive patterns for purchase cycles

in over 100 product categories

In each category, build models to describe

behaviour most likely to lead to an ad response

(i.e. click).

Score each user for fit with every

category…daily.

Target ads to users who get highest ‘relevance’

scores in the targeting categories

Varying Product Purchase CyclesMatch Users to the ModelsRewarding Good BehaviourIdentify Most Relevant Users

28


Recency Matters and So Too Intensity

Active now… …and with feeling

29


Differentiation | Category specific modelling

time

inte

nsi

ty s

core

time

inte

nsi

ty s

core

Inte

nse

Cli

ck Z

one

Example 1: Category Automotive Example 2: Category Travel/Last Minute

Different models allow us to weight and determine intensity and recency

Alt Behaviour 1: 5 pages, 2 search keywords, 1 search click, 1 ad click Alt Behaviour 1: 5 pages, 2 search keywords, 1 search click, 1 ad click

Inte

nse

Cli

ck Z

one

30


Differentiation | Category specific modelling

time

inte

nsi

ty s

core

Intense Click Zone

Example 1: Category Automotive

Different models allow us to weight and determine intensity and recency

with no further activity, decay takes effect

Alt Behaviour 1: 5 pages, 2 search keywords, 1 search click, 1 ad click

user is in the Intense Click Zone

31


Audiences Defined by Interests and Actions

We found:

1,900,000 people looking for

mortgage loans.

+122%

CTR Lift

Mortgages Home Loans Refinancing Ditech

Financing section in Real Estate

Mortgage Loans area in Finance

Real Estate section in Yellow Pages

+626%

Conv Lift

Example search terms qualified for this target:

Example Yahoo! Pages visited:

Source: Campaign Click thru Rate lift is determined by Yahoo! Internal

research. Conversion is the number of qualified leads from clicks over number of impressions served. Audience size represents the audience within this behavioral interest category that has the highest propensity to engage with a brand or product and to click on an offer.

Date: March 2006

Results from a client campaign on Yahoo!

NetworkExample: Mortgages

32


Products: Retargeting

• Target customers based on any past event

• Lead consumers down the purchase funnel

• Can be quite clever creatively

– Improving offers

• There are time limitations

Next visit on Yahoo!

An event

e.g. People who viewed a certain

ad

Second ad with additional

information increases awareness,

engagement!

Retargeting

33


Example: Retargeting – BMW in Germany

BMW ran a homepage event in Germany

Can re-target those who saw the event, clicked on the event or

searched for BMW or a combination

Can increase conversion


Brand Ads and Search Ads Interact!

• Is ad search strategy enough for a direct marketer?

• Do brand ads play a role in search advertising?

• Harris Direct Case Study

Awareness

Purchase

Consideration

35


Case Study: Harris Direct

On:

Viewing These Ads: Had This Effect On:

• Brand Favorability– Up 32%

• Purchase Intent– Up 15%

• Aided Brand Awareness– Up 7%

36

http://finance.yahoo.com/?u

http://my.yahoo.com/

http://ads.pointroll.com/DefaultAd/?ad=058h200461015355&pub=yhoofin&size=300_250&click=1&code=no&scr=no

http://my.yahoo.com/


Case Study: Harris Direct

People who saw display ads were 61% more likely to search on related topics…

…and drove 139% more clicks on

algorithmic and sponsored links…

…specifically driving 249% more

sponsored search clicks …

…and driving 91% more activity

on the HarrisDirect.com website.

37

Personalization Case-Study

Overstock.com Case Study


Brand LoyaltyPersonalized

Brand Experience

Personalized Home Page

Brand Advertising

Personalization Effects on Consumer Marketing

Across All Retail Channels

Personalization Drives Performance Everywhere

SalesPersonalized Up Sells

& Cross-sells

Product Detail

Shopping Cart

Order Confirmation

RetentionPersonalized

Re-Targeting

Display Ads + Landing Page

Email & Subject Line Offers

AcquisitionRelevant

Advertising

Landing Pages

Display Advertising

Banner Affiliates

Call Center & In-Store Kiosk

Product Recommendations


Targeted AdvertisingPersonalized Ad Delivery

Results:

550% higher

revenue per

impression

Consumer Value Drives Business Performance

RecommendationsDiscovery, Serendipity

Results:50% longer

rental queues

CommerceOptimized „Next Sell‟ & „Cross Sell‟

Results:250% more sales

per visit

Direct MarketingTargeted Mail & Promotions

Results:30% Increase in

VOD sales

CommunityLike-Minded Buddies& Social Groups

Results:

300% longer

subscriber

lifetime

Product Promotions„More Like This…‟ Complements

Results:

400% ROI

in 5 mos.

40


Marcia• Bought comforter last week

• Registered for baby items

• Abandoned cart with coat

Fran• Recently bought a coat

• Lately browsing jewelry

• Added shoes to wish list

Overstock Showed Every User the Same Ad


Marcia• Bought comforter last week

• Registered for baby items

• Abandoned cart with coat

Fran• Recently bought a coat

• Lately browsing jewelry

• Added shoes to wish list

ChoiceStream Personalizes Each Ad…


FranClicks on product in ad

Personalized Landing PageContains more product recommendations

…Giving Shoppers a Compelling “Call to Purchase”


Overstock Cares About ChoiceStream RealRelevance®

Advertising Lift*

Conversions

per Thousand Impressions Over 3x

Revenue

per Thousand Impressions 3x-5x

Improving the Metrics that Matter Most

* Source: Actual results based on A/B tests comparing advertiser's best-performing display ads against ChoiceStream

ads with personalized product recommendations.

Open Insights

and the wisdom of the crowds...

Social Media

Research


45


Evolution of Social Media

• Although the “traditional notion” of portal and web content is still attracting growing audiences

• The original notion of “publishing content” to attract audiences is changing fast

– As people discover the fact that the Internet is an Interactive Medium

– The uses of the Internet enter areas we could not imagine a short time ago

• A new notion of “publishing” is fast emerging

– The opportunity of user-generated content

46


Examples of Social Media

• Blogs

– The individual as publisher

– Comments and tags part of the process

• Sharing Photos: e.g. Flickr

• Social Search

– My Web 2.0

– Yahoo! Answers

– Del.icio.us

• Web communities:

– Yahoo! Groups

– Individual web presence: Facebook, MySpace, Yahoo! 360, Friendster, …

• Video sharing: You Tube, Yahoo! Video, etc…

47

Copyright 2009 by U.M. Fayyad, All rights reserved www.open-insights.com48


What makes Flickr special?

1. User Generated Content

Content not licensed from providers such as Corbis or Getty, but rather contributed by

users.

49



2. User Organized Content

Content is tagged, described, organized, discovered, etc. not by “editors” but by the

users themselves.

50




3. User Distributed Content

Flickr achieved distribution across the internet, not through “business deals” per se, but

rather through the Flickr community which distributed Flickr content on 3rd-party blogs.

52



4. User Developed Functionality

Flickr exposed APIs (PHP, Perl, etc.) that allowed the community of developers to build

against the Flickr platform.

53



1. User Generated Content

Content not licensed from providers such as Corbis or Getty, but rather contributed by

users.



users themselves.

3. User Distributed Content

Flickr achieved distribution across the internet, not through “business deals” per se, but

rather through the Flickr community which distributed Flickr content on 3rd-party blogs.

4. User Developed Functionality

Flickr exposed APIs (PHP, Perl, etc.) that allowed the community of developers to build

against the Flickr platform.

Entire ecosystem created by less than ten employees…

aided by millions in the Flickr community.

54


New Science?

• The Internet touches all of our lives: personal, commercial, corporate, educational, government, etc…

• Yet many of the basic notions we talk about:

– Search, Community, Personalization, Engagement, Interactive Content, Information Navigation, Computational Advertising

– Are not at all understood, or well-defined

– These are not disciplines that academia or any industry research labs focus on…

55


Areas of Research

• Community:– How do you know what to believe on the Internet?

– Trust models on-line and trust propagation

– What makes communities thrive? Whither?

– Social media, tagging, image and video sharing

• Microeconomics: a new generation of economics driven by massive interactions– Auction marketplaces

– The web as a new LEI of activities and economies

• Information Navigation and Search– We are in the early days of search and retrieval

• Computational Advertising

56

Open Insights

“Wisdom of Crowds”

Is the Turing test always the right question?

57


A Digression:Computer Vision is hard

58

Copyright 2009 by U.M. Fayyad, All rights reserved www.open-insights.com59 Luis Von Ahn, Carnegie-Mellon University (Google Image)









users themselves.

64








Challenges in social media

• How do we use these tags for better search?

• What‟s the ratings and reputation system?

• How do you cope with spam?

• The bigger challenge: where else can you exploit the power of the people?

• What are the incentive mechanisms?

71

Open Insights

Illustrating New Directions on Yahoo! Network

Case Study: Social Targeting

Research


72


What is Social Targeting?

Social Targeting (ST) extends individual targeting techniques by considering

individual characteristics of a user’s close social contacts.

Individual Targeting + Social Networks = Social Targeting

Demographic

Geographic

Behavioral

Contextual

+ = Demographic

Geographic

Behavioral

Contextual

All of the above data

for friends

&

73


Why use Social Targeting?

Illustrative Social Graph People demonstrating interest in

Football by playing Yahoo Fantasy

Football

Social Targeting provides indications of unobservable

and off-network interest categories.

Friends with shared interest in

Football, but who don’t play Fantasy

Football at Yahoo

74


Why does Social Targeting work?

Diffusion

People within a group are exposed to similar ideas

Homophily

People seek out others with similar interests

Social Identity

Cultural preferences signal group memberships

People share similar interests and

preferences with their close social contacts.

75


Yahoo! data for Social Targeting

Instant Messenger

Fantasy Sports

Yahoo!’s social networks can help improve monetization and user experience

through better ad/content targeting.

76


Empirical Tests Demonstrate Value of ST

Targeting Ads at custom lists of users based on Social Extension increases

inventory without decreasing CTR

Instant Messenger

Inventory Expansion

from Seed Population

+483% from sample of

BT Weight Loss

+68% from sample of

BT Food & Nutrition

+36% from sample of

Y! Fantasy Football users

CTR of Social Extension vs

Seed Population

117% vs sample of

BT Weight Loss

104% vs sample of

BT Food & Nutrition

107% vs sample of

Y! Fantasy Football users

77


Case Study: TWITTER Social Marketing?

Diffusion

Give 20 free DVD’s to major related artists/groups, ask them

To notify Twitter groups – reached over 2M people

Marketing ANVIL movie

Very niche audience, how do you reach them?

Social Identity: power of word of mouth...

What is the Cost to VH-1?Compare with traditional approach: TV commercials to promote a

documentary film?

Viacom’s VH1 Twitter campaign on ANVIL (the movie)(week of May 14th , 2009 – see AdAge article)

78

Open Insights

Many, many challenges...

What Does this all Mean for Search?

Research


79


Content trends

[Ramakrishnan and Tomkins 2007]


Metadata trends

[Ramakrishnan and Tomkins 2007]

Open Insights

Towards Getting Things Done… vs. Searching

Research 82



Example

I want to book a vacation in Tuscany.Start Finish

83


Loved the vacation, want to make that sweet Italian coffee at home

85


Trends in task complexity

• Dawn of search:

– Navigational queries

– Pockets of information

• Today:

– Increasing migration of content online

– New forms of media only available online

– Infrastructure for payments and reputation sufficient for many users

86


Things to notice

• Long-running user goals

• Search as hub:

– start there

– return for resource discovery and at task boundaries

– traverse the web broadly to complete task

• Web services integrated into task

87


What does this mean to Search?

• Publishers and search engine collaborate

• Users see richer search experience

• Accomplish their tasks faster and more effectively

• Example: abstracts surfacing structured content

• Back to the Future?


babycenter

epicurious

An Example: Search results in Search Monkey

yelp.com

answers.com

LinkedIn

webmd

Gawker

New York Times


An Example: Search results in Search Monkey

yelp.com

babycenter

epicurious

answers.com

LinkedIn

webmd

New York Times

Gawker


What are the challenges?

• Community of users

– Social system

• Incentives and reputations

– Economic system

• Poorly phrased, grammatically limited queries

– Language analysis

• Improving user experience from past data

– Data mining


What we did not cover today...

• Applications of On-line Data for Enhanced Prediction

– Product Sales (e.g. Car Manufacturer)

– Customer Churn in Telco

– Banking and Financial Services

– Retail Cross-sell and Up-Sell

• The new generation of On-line Marketing Solutions

based on the evolution of Search

– Task orientation

– Long-running queries

92


Experience summary at Yahoo!

• Dealing with the largest data source in the world (25 Terabyte

per day)

• Search is in its early incarnations still... where will it evolve to?

We don’t know

– Verticalization trend

– Solving tasks not retrieving pages

– Back to the future: editorial input and directory V2.0?

– Objects and entity extraction

• We cannot separate Search from Search Advertising

– Think of the two search engines as intertwined and

inseparable

93


Lessons Summarized• Evolution from Search and Analytics to Consumer relevance

and advertising

• On-line marketing is ripe with opportunities for predictive

analytics applications

• Results are very powerful and we lack a true understanding

• New generation using social media data and more advanced

predictive analytics are very promising

• Combining context analysis (including text analytics) with user

modeling/targeting leads to some very powerful advances in

relevance

94

Open Insights

[email protected]

Thank You! & Questions?


95

Documents

Predictive Analytics over On-line and Social Network Data · 2010-06-08 · Predictive Analytics over On-line and Social Network Data Usama Fayyad, Ph.D. CEO, Open Insights ... Spring