1

Interpreting Advertiser Intent in Sponsored SearchBHANU C VATTIKONDA, SANTHOSH KODIPAKA, HONGYAN ZHOU, VACHA DAVE, SAIKAT GUHA, ALEX C SNOEREN

2

Organic results

Sponsoredresults

Query

3

women’s shoes

Ad Keyword

shoesQuery

4

Huge and growing industry

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 20140

5

10

15

20

25

30

Search Revenues ($ billions)

20%

5

What is today’sdate in Japan

Query

Date JapanAd Keyword

6

Virgin River Utah

Query

Virgin MediaAd Keyword

7

Virgin River Utah

Query

Virgin MediaAd Keyword

We use organic search results for ad keyword

8

Capturing user and advertiser intents

OverviewMismatching advertiser and user intent

Organic results are accurate

Ad keywords capture advertiser intent

Ad keyword is very brief

Use organic results for ad keyword to interpret advertiser intent

9

10

Relevance in Sponsored Search

11

Ad selection pipeline

Ad Corpus(M’s of ads)

UserQuery

• Quick look up • “virgin media”

matched to “virgin river Utah”

• Choose ads that provide most revenue

• Measure the relevance of each ad to user query

• Virgin media ad seen as relevant to “virgin river Utah”

Ad retrieval Relevance Auction

12

Supervised machine learning

Training

Training set(query, ad, judgement)

Trained Ranker

Compute numerical featuresE.g., no. of words common in query and ad keyword

(features, judgement)

Ranker can compute relevance of (query, ad) pair

13

Signal sources

Virgin River UtahQuery

Virgin MediaAd Keyword

14

Signal sources

Virgin River UtahQuery

Virgin MediaAd Keyword

Ad Creative

15

Signal sources

Virgin River UtahQuery

Virgin MediaAd Keyword

Ad Creative

Landing page

16

Features from signal sourcesQuery and ad creative

Query: “virgin river Utah”, ad title: “virgin media | virginmedia.com”

Word bigram overlap: 0

Query and landing page

Query: “virgin river utah”, landing page title: “Virgin Media – Cable Broadband,

Digital TV”

Ordered word bigram overlap: 0

Query and ad keyword

Query: “virgin river Utah”, ad keyword: “virgin media”

Word unigram overlap: 0.5

17

Interpreting user and advertiser intent Query and ad keyword are very short -- 2.5 words on average

Hard to determine user and advertiser intent

Query and ad keyword may mean same without overlap or vice versa

Sneakers vs shoes

Virgin river Utah vs virgin media

Use organic search results to boost query and ad keyword

18

Matching user and advertiser intents

50% overlap

25% overlap3% overlap

19

Improving relevance rankerIntroduce features capturing overlap between user and advertiser intents

User intent captured using search results for user query

Advertiser intent captured using search results for ad keyword

75 Features introduced:

Overlap between elements of search results

Overlap between ad creative and search results for ad keyword

20

Evaluation

21

Measuring performance of ranker

Trained Ranker

Validation set(query, ad, judgement)

(query, ad, relevance)

Compare ranker result against human judgement

Compute numerical features

(query, ad, judgement)

22

Evaluating performance of ranker Did the ranker identify most of the relevant results?

How accurately did the ranker identify relevant ads?

23

Precision-RecallLow precision results in irrelevant ads being shown to users

Bad user experience

Wasteful spend for advertisers

Low recall would lead to missed chances

Lost revenue opportunities for search engine

Lost targeting for advertisers

Lost desirable ads for users

24

Dataset and approachData from a large search engine

1.28M (query, ad) pairs of training data

320k (query, ad) pairs of validation data

Several hundred existing features

The ranker is trained on a combination of the features we introduce and existing production system features

Compare new ranker to current ranker

25

Query and ad matching Query and ad matched using different match types chosen by advertiser

Exact match

Ad keyword: “shoes” and query: “shoes”

Broad match

Ad keyword: “shoes” and query: “sneakers”

Features we introduce capture similarity between query and ad keyword

26

Significant improvement in broad match

2.7% improvement in area under precision-recall curve

27

SummaryMismatched user and advertiser intents leads to errors in sponsored ads

Interpreting ad keywords poses a challenge due to their brevity

Organic search results for the ad keyword capture advertiser intent

2.7% gain in area under precision-recall curve over production

28

Thanks

29

Thanks

30

Backup

Organic results vs ad results Organic and ad result goals are different

Ads “related” to user query

Example:

User query: “prom dresses”

Ad for Limousine service targeting “prom dresses”

Good ad but bad organic result

8

Ad keywords are for targeting