On Search, Personalisation and Real-time Advertising

On Search, Personalisation and Real-time Advertising

Dr. Jun Wang, Senior Lecturer Computer Science, University College London

Email: [email protected] Twitter: @seawan

30/10/14 Dunnhumby Talk 1

Summary


Web search Ads (display opportunities) - Maximize profit

Search results - Maximize users’ satisfactions?

Query


Recommder Systems


Recommeder Systems


Kruschwitz, Udo <[email protected]>

Real-‐2me Adver2sing


7


“This is Lawrence from India. I was searching Recommender model in web and found your webpage in search engine. Then, I visited your webpage searching relevant contents and saw unrelevant Google add in "Research Team" page (aFached screenshot). This add might vary from country to country. But I feel it will mislead and give wrong opinion to users who visit your webpage.” -‐ Lawrence from India

Summary


Search (Informa2on Retrieval)

l  General definiJon: search large-‐scale unstructured data, mostly text documents, but also include images, videos, etc

l  ApplicaJons: –  web search –  product search –  enterprise search –  desktop/email search –  informaJon filtering –  collaboraJve filtering and recommeder systems


Queries can have ambiguous intents

[Courtesy of F. Radlinski, MSR Cambridge]

Columbia clothing/sportswear

Colombia (Country: misspelling)

Columbia University

Columbia Records music/video columbia


Diversified search results

Diversifica2on -‐> nega2ve correla2on -‐> reduce the risk: see our sigir09 paper on porQolio theory of informa2on retrieval


Recall driven personalised search: relevance feedback revisit

•  www13 paper

Figure 1: Example application, where Page 1 contains the diversified, exploratory relevance ranking, and

Page 2 contains a refined, personalised re-ranking of the next set of remaining documents, triggered by

the “Next” button and depending on the documents viewed on Page 1 (the left Page 2 ranking contains

documents about animals, the right ranking documents about cars). The same search process can continue

in the remaining results pages until the user leaves.

ing the search experience. Interaction can be split into twocategories, system driven and user driven. In system driveninteraction, the system directly involves the user in the in-teractive process; for instance, a user can narrow down ane-commerce search by filtering based on item attributes thesystem has intentionally specified [42]. Somewhat related isthe field of adaptive filtering, where a user refines an infor-mation stream over time by giving feedback [23].

User driven interactions can allow the user to opt-in, orremain unaware of the interaction due to the use of implicitfeedback, which can come from a number of sources, in-cluding scrolling behaviour, time spent on search page andclickthroughs, the latter two of which correspond well withexplicit user satisfaction [13]. Due to their abundance andthe low cost of recording them, clickthroughs have emergedas a common measure of implicit feedback, with much re-search into using search click logs for learning ranking algo-rithms [15].

When implicit feedback is used in interactive retrieval, theresearch can be split into long term and short term objec-tives. Long term interactive retrieval methods are usuallyconcerned with search personalisation or learning to rankover time, for instance, incorporating user behavior fromsearch logs into a learning to rank algorithm [1], re-rankingsearch results for specific users by accumulating user dataover time [33] and using clickthroughs to learn diverse rank-ings dynamically over time [20]. Such techniques are usefulfor improving a search system over time for popular queries,but are less able to deal with ad hoc web search and tailqueries. On the other hand, short term methods aim toimprove the immediate search session by incorporating usercontext such as clickthroughs and search history [28], butmay have issues with user privacy concerning using userdata.

In this paper, we propose a technical scheme for shortterm, interactive ranking that is able to use user feedbackfrom the top-ranked documents in order to generate con-textual re-rankings for the remaining documents in ad hocretrieval. The feedback could be collected either implicitly,such as from clickthroughs and search page navigation, orexplicitly, such as from user ratings. The proposed feedbackscheme naturally makes use of the fact that in many com-

mon retrieval scenarios, the search results are split into mul-tiple pages that the user traverses across by clicking a “nextpage” botton. A typical use case would be a user examin-ing a Search Engine Results Page (SERP) by going throughthe list, clicking on documents (e.g., to explore them) andreturning to the SERP in the same session. Then, whenthe user indicates that they’d like to view more results (i.e.the ‘Next Page’ button), the feedback elicited thus far isused in a Bayesian model update to unobtrusively re-rankthe remaining documents (which are not seen as yet), onthe client-side, into a list that is more likely to satisfy theinformation need of the user, which are shown in the nextSERP, as illustrated in Figure 1.We mathematically formulate the problem by considering

a Bayesian sequential model; we specify the user’s overallsatisfaction over the Interactive Exploratory Search processand optimise it as a whole. The expected relevance of doc-uments is sequentially updated taking into account the userfeedback thus far and the expected document dependency.A dynamic programming approach is used to optimise thebalance between exploration (learning the remaining docu-ments’ relevancy) and exploitation (presenting the most rel-evant documents thus far). Due to the optimisation calcu-lation being intractable, the solution is approximated usinga range of techniques such as monte carlo sampling and asequential ranking decision rule. Our formulation requiresno changes to the search User Interface (UI) and the updatestage can be performed on the client side in order to respectuser privacy and deal with computational e�ciency. We testthe method on TREC datasets, and the results show thatthe method outperforms other strong baselines, indicatingthat the proposed interactive scheme has significant poten-tials for exploratory search tasks.The remainder of the paper is organised as follows: in Sec-

tion 2 we continue our discussion about the related work; inSection 3 we present the proposed dynamical model. Itsinsights and its approximate solutions are presented in Sec-tions 4 and 5 respectively. The experiments are given inSection 6 and conclusions are presented in Section 7.

Xiaoran Jin, Marc Sloan, and Jun Wang. InteracJve Exploratory Search for MulJ Page Search Results, www13

exploratory relevance ranking Personalised re-‐ranking


Recall driven search: relevance feedback revisit

•  www13 paper

Figure 1: Example application, where Page 1 contains the diversified, exploratory relevance ranking, and

Page 2 contains a refined, personalised re-ranking of the next set of remaining documents, triggered by

the “Next” button and depending on the documents viewed on Page 1 (the left Page 2 ranking contains

documents about animals, the right ranking documents about cars). The same search process can continue

in the remaining results pages until the user leaves.

ing the search experience. Interaction can be split into twocategories, system driven and user driven. In system driveninteraction, the system directly involves the user in the in-teractive process; for instance, a user can narrow down ane-commerce search by filtering based on item attributes thesystem has intentionally specified [42]. Somewhat related isthe field of adaptive filtering, where a user refines an infor-mation stream over time by giving feedback [23].

User driven interactions can allow the user to opt-in, orremain unaware of the interaction due to the use of implicitfeedback, which can come from a number of sources, in-cluding scrolling behaviour, time spent on search page andclickthroughs, the latter two of which correspond well withexplicit user satisfaction [13]. Due to their abundance andthe low cost of recording them, clickthroughs have emergedas a common measure of implicit feedback, with much re-search into using search click logs for learning ranking algo-rithms [15].

When implicit feedback is used in interactive retrieval, theresearch can be split into long term and short term objec-tives. Long term interactive retrieval methods are usuallyconcerned with search personalisation or learning to rankover time, for instance, incorporating user behavior fromsearch logs into a learning to rank algorithm [1], re-rankingsearch results for specific users by accumulating user dataover time [33] and using clickthroughs to learn diverse rank-ings dynamically over time [20]. Such techniques are usefulfor improving a search system over time for popular queries,but are less able to deal with ad hoc web search and tailqueries. On the other hand, short term methods aim toimprove the immediate search session by incorporating usercontext such as clickthroughs and search history [28], butmay have issues with user privacy concerning using userdata.

In this paper, we propose a technical scheme for shortterm, interactive ranking that is able to use user feedbackfrom the top-ranked documents in order to generate con-textual re-rankings for the remaining documents in ad hocretrieval. The feedback could be collected either implicitly,such as from clickthroughs and search page navigation, orexplicitly, such as from user ratings. The proposed feedbackscheme naturally makes use of the fact that in many com-

mon retrieval scenarios, the search results are split into mul-tiple pages that the user traverses across by clicking a “nextpage” botton. A typical use case would be a user examin-ing a Search Engine Results Page (SERP) by going throughthe list, clicking on documents (e.g., to explore them) andreturning to the SERP in the same session. Then, whenthe user indicates that they’d like to view more results (i.e.the ‘Next Page’ button), the feedback elicited thus far isused in a Bayesian model update to unobtrusively re-rankthe remaining documents (which are not seen as yet), onthe client-side, into a list that is more likely to satisfy theinformation need of the user, which are shown in the nextSERP, as illustrated in Figure 1.We mathematically formulate the problem by considering

a Bayesian sequential model; we specify the user’s overallsatisfaction over the Interactive Exploratory Search processand optimise it as a whole. The expected relevance of doc-uments is sequentially updated taking into account the userfeedback thus far and the expected document dependency.A dynamic programming approach is used to optimise thebalance between exploration (learning the remaining docu-ments’ relevancy) and exploitation (presenting the most rel-evant documents thus far). Due to the optimisation calcu-lation being intractable, the solution is approximated usinga range of techniques such as monte carlo sampling and asequential ranking decision rule. Our formulation requiresno changes to the search User Interface (UI) and the updatestage can be performed on the client side in order to respectuser privacy and deal with computational e�ciency. We testthe method on TREC datasets, and the results show thatthe method outperforms other strong baselines, indicatingthat the proposed interactive scheme has significant poten-tials for exploratory search tasks.The remainder of the paper is organised as follows: in Sec-

tion 2 we continue our discussion about the related work; inSection 3 we present the proposed dynamical model. Itsinsights and its approximate solutions are presented in Sec-tions 4 and 5 respectively. The experiments are given inSection 6 and conclusions are presented in Section 7.


Exploratory ranking Personalised re-‐ranking


Recall driven search: relevance feedback revisit

•  We consider MulJ Page Search Results •  Intend to opJmise overall expected effecJveness over the search journey

•  Our derivaJon shows that –  Page 1 contains the diversified, exploratory relevance ranking

–  Page 2 contains, personalised re-‐ranking of the next set of remaining documents, where the relevance feedback is triggered by the “Next” bugon

certain objective function. This rank action is denoted asvector s

t = [st1

, . . . , stM ] where stj is the index of the docu-ment retrieved at rank j at the tth research page i.e., s1 isthe page 1 (initial) ranking and s

2 is the page 2 (modified)ranking. We let s represent all rank actions s

1 . . . sT . Wedenote r = [r

1

, . . . , rK ] as the vector of feedback informa-tion obtained from the user for a given page, where K is thenumber of documents given feedback with 0 K M , andri is the feedback information gained (the rating providedby the user) of relevance feedback for document i, either bymeasuring a direct rating or by observing clickthroughs.We use a weighted sum of the expected DCG@M scores of

the rankings of the T upcoming result pages, denoted hereby (note that Rstj

⌘ Rtstj)

Us =TX

t

0

@�t

tMX

j=1+(t�1)M

E(Rstj)

log2

(j + 1)

1

A (2)

to represent the user’s overall satisfaction, where E(Rstj) =

✓stj is the expected relevance of a document at rank j in

result page t. We have chosen the objective function as itis simple and both rewards finding the most relevant docu-ments and also ranking them in the correct order, althoughother IR metrics can be adopted similarly. The rank weight

1

log2 jis used to give greater weight to ranking the most rele-

vant documents in higher positions. The tunable parameter�i � 0 is used to adjust the importance of result pages andthus the level of exploration in the initial page(s). When �

1

is assigned to be zero, the initial ranking is chosen so as tomaximise the next page(s) ranking and thus the priority ispurely on exploration [39]. It could also be a decay functionreflecting the fact that less users are likely to reach the laterresult pages. An extreme case is that when �t = 0, 8t > 1,there is no exploration at all and the documents are rankedaccording to the PRP. In this paper, we fix �t a priori, whileleaving the study of adaptive �t according to learned infor-mation intents [3, 21] for future work.

T indicates how far ahead we consider when ranking doc-uments in the current page. Our experiments using TRECdata indicate that setting T = 2 (only considering the im-mediate next page) gives the best results. To simplify ourderivations, we consider T = 2 for the remaining derivationswhile the resulting algorithms can be easily generalised tothe case where T > 2. Thus, our overall objective is to findthe optimal rank action s

⇤ such that U is maximised

s

⇤ = argmaxs

⇢�

MX

j=1

✓s1jlog

2

(j + 1)

+ (1� �)2MX

j=1+M

✓s2jlog

2

(j + 1)

�(3)

where we use � to balance the importance of pages 1 and2 when T = 2. The statistical process is represented by anInfluence Diagram as shown in Figure 2.

3.1 Relevance Belief UpdateWe start by considering the problem in reverse; given that

we’ve already displayed a ranking to the user, how can weuse their feedback to update our ranking in the second resultpage. After receiving some observation of feedback ri, andwith the user requesting more documents, we can set the

θ,Σ θ’,Σ’

U1 U2

s1 r s2

Figure 2: The interactive search process as illus-

trated by an influence diagram over two stages. In

the diagram, the non-filled circular nodes represent

random variables;the rectangular nodes represent

the rank action at each result page. The rhombus

node is the instant utility.

conditional distribution of the probability of relevance giventhe feedback as

P (R2|r) / P (R2, r)

= P (R2

1

, R2

2

, . . . , R2

N , Rs11= rs11

, . . . , Rs1k= rs1k

) (4)

where rs1jis the observed feedback on the document re-

trieved at rank j and k is number of documents with user’sjudgements after the first result page.If we reorder the vector R and ✓ as

R =

✓R\s1

Rs1

◆,✓ =

✓✓\s1✓s1

◆, (5)

where \s1 means excluding documents which have feed-back after the first result page, ⌃ becomes the covariancematrix of the new R, given by

⌃ =

✓⌃

11

⌃12

⌃21

⌃22

◆(6)

Then, we can specify the conditional distribution

P (R2|r) / exp(�12(

✓R\s1

r

◆� ✓)T ·⌃�1 · (

✓R\s1

r

◆� ✓))

(7)

giving us the following posterior update functions

R

2 ⇠ N (✓2,⌃2) (8)

✓2 = ✓1

\s1 + ⌃12

⌃�1

22

(r� ✓1

s1) (9)

⌃

2 = ⌃11

� ⌃12

⌃�1

22

⌃21

(10)

These formulas can then be used to generate new, pos-terior relevance score estimates for the documents, on theclient-side, based on the feedback received in result page 1.These score updates can be used to re-rank the documentsin result page 2 and displayed to the user.

3.2 User FeedbackBefore presenting a ranked list at result page 1 and di-

rectly observing r, it remains an unknown variable and canbe predicted using our prior knowledge. In order to per-form exploration (active learning), we will need to be able



How it works

30/10/14 Dunnhumby Talk

x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤

x x doc about apple ceo

X: doc about apple fruit

¤

¤ O: doc about apple iphone

Page 1: diversified, exploratory relevance ranking considers Relevancy + Variance + |CorrelaJons| Page 2: personalised re-‐ranking

16

How it works


x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤



¤


Page 1: diversified, exploratory relevance ranking considers Relevancy + Variance + |CorrelaJons|

Q

17

How it works


x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤



¤



Q

18

How it works


x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤



¤



Q

-‐1

-‐1

+1

19

How it works


x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤



¤


Page 2: Personalised reranking:

Q

-‐1

-‐1

+1

Q

20

How it works


x x

x x

x x x x

x x

x o o o o

o

o

o

¤

¤ ¤

¤

¤

¤

¤ ¤



¤


Page 2: Personalised reranking:

Q

-‐1

-‐1

+1

Q

21

Summary


Is Personalized Rec. Always BeFer? Non-‐personalized

Top Ar2sts in October Personalized:

Ar2sts Recommended for You

Personalized vs Non-‐Personalized

Dataset: Movielens-‐100k

Personalized vs Non-‐Personalized

•  Personalized top-‐N CF as a learning model –  Improve the object of overall relevance – But does NOT improve on each user

POP BPR

Top-‐N Performance Low bias

High variance

Switch between Two PorQolios

Porqolio DominaJon

Case Study 1

•  Demography: 50 years old male programmer •  History: 86 feedbacks, most of which are unpopular items

Case Study 1

•  Analysis: BPR porqolio dominates POP porqolio •  Results: BPR has beger ranking performance

Case Study 2

•  Demography: 28 years old male engineer •  History: 15 feedbacks, most of which are popular items

Case Study 2

•  Analysis: POP porqolio dominates BPR porqolio •  Results: POP has beger ranking performance

Cold-‐start problem in recommmender systems

Interac2ve Recommender Systems

Possible Solu2ons

Zhao, Xiaoxue, Weinan Zhang, and Jun Wang. "InteracJve collaboraJve filtering." CIKM, 2013.

Objec2ve

Cold-‐start problem Interac2ve mechanism for CF


Proposed EE algorithms

Thompson Sampling

Linear-‐UCB

General Linear-‐UCB


Cold-‐start users


Summary



Life of a display ad in the RTB environment: 0.36 seconds

39

Ad Exchange

Demand-Side Platform

Advertiser

Data Management

Platform

0. Ad Request 1. Bid Request (user, context)

2. Bid Response (ad, bid)

3. Ad AucJon 4. Win NoJce (paying price)

5. Ad (with tracking)

6. User Feedback (click, conversion, etc.)

User InformaJon

User Demography: Male, 25, Student, etc. User SegmentaJons: Ad science, London, etc.

Webpage

User

DSP (Demand Side PlaQorm)

30/10/14

Bidder in DSP

Op2mal Bidder: Problem Defini2on

Bid Engine Bid Request Bid Price

42

Input: bid request include Cookie informaJon (anonymous profile), website category & page, user terminal, locaJon etc Output: bid price Considera2ons: Historic data, CRM (first party data), DMP (3rd party data from Data Management Plaqorm)

What is the op2mal bidder given a budget constraint? e.g., Maximise Subject to the budget constraint

43

The General Process for Bidding Op2misa2on

Red: hard constraints Green: features Blue: models Note that “Frequency & recency rules” are also used as features

Op2mal bidder: the formula2on •  FuncJonal OpJmisaJon Problem

–  Dependency assumpJon:

•  SoluJon: Calculus of variaJons

context+ad features

CTR esJmaJon winning funcJon

bidding funcJon

Weinan Zhang, Shuai Yuan, Jun Wang, OpJmal Real-‐Time Bidding for Display AdverJsing, KDD’14

Op2mal bidder: the solu2on


Experiments

Offline Online

Winner of the first global Real-‐Jme Bidding algorithm contest 2013-‐2014


UCL OpenBidder Benchmarking System

47

(RTB) Ads prices are vola2le

The price movement of a display opportunity from Yahoo! ads data Under GSP (generalized second price aucJon)

Selling Futures Online Advertising Slots viaOption Contracts

Jun Wang and Bowei Chen{j.wang, bowei.chen}@cs.ucl.ac.uk

Department of Computer Science, University College London, UK

§1 MotivationAffected by the dual force of supply and demand, the agreed prices

of an online ad slot often exhibit abrupt and extreme changes over

time (see Figure 1). This makes the cost of displaying ad slots (for

advertisers) and the advertising incomes (for publishers and search

engines) unpredictable. Thus there are increasing needs of a new

advertising trading mechanism to manage the risk of cost or income.

(a)

Jun/2002 Jan/2003 Jun/20030

10

20

30

40

50

Time

Pric

e (U

S $)

Ad slot price (GSP)

(b)

Jan/2003 Jun/2003−1

−0.5

0

0.5

1

1.5

Time

Pric

e (U

S $)

Price change rate

(c)

Figure 1: Plots of a selected Yahoo! online advertisement from 2002 to 2003: (a) advertisers’

bidding prices; (b) accepted generalized second-prices (GSPs); (c) change rates of the GSPs.

§2 Ad Option and Its Pricing

An Ad Option is defined as a contract signed by advertisers

and publishers or search engines. The ad option can give its

buyer (normally advertisers) the right to buy a certain amount of

advertising impressions or clicks from its seller (normally publishers

or search engines) in the future time at a prefixed price. The ad

option buyer needs to pay its seller an upfront fee at the current

time, called Ad Option Price.

There are two benefits of the ad option contract: 1) it is a new

advertising trading mechanism that bridges the spot and the future

markets together; 2) it can help advertisers, publishers and search en-

gines to manage their future cost and incomes smoother, less volatile

and be easier to predict.

A key question is:“How much of an ad option price shouldbe?" Let us consider an example of ad option pricing for impression-

based ad slots. In the future time, assume there are total M impres-

sions to sell and two movement directions of ad slot price, say, Aut+�tand Adt+�t. The publisher would like to sell ↵M impressions via ad

options and sell (1� ↵)M impressions on the spot markets.

Figure 2: The binomial tree framework for the ad slot price movement.

The future ad slot price can be estimated from forecasting models,

i.e. E(At+�t) and var(At+�t), where � in the Figure 2 is an estimation

parameter. The future incomes of publisher are assumed to be risk-

less. That is, we let the upside and downside incomes equal and

obtain the number of the ad options to sell. To avoid the arbitrage,

the future advertising income is expected to be at least equal to the

current income from the spot markets. Based on that we then obtain

the fair ad option price.

§3 Experiments and ConclusionIn the experiments we have used Yahoo! online advertising dataset

and have investigated Geometric Brownian motion and mean-reverting

process to describe the movement of ad slot prices. Figure 3 shows

that � = 1.3 gives the smallest mean value of RMSEs, so we choose

it as the optimal default value. Table 1 then illustrates clearly that the

publisher’s risk is reduced substantially when he increases the number

of ad options from zero to the number of ad options suggested by

the pricing model (↵M/1000).

0 0.5 1 1.5 25

10

15

20

λ

Mea

n of

RM

SEs

(US

$)

Mean of RMSEs

Figure 3: The impact of parameter � on the accuracy of estimated ad option prices.

Table 1: Risk vs. reward of a Yahoo! online ad.

Time 0 options

12↵M1000 options

↵M1000 options

5 294.6 (116.5) 263.2 (67.2) 231.8 (18.9)

10 313.3 (104.4) 284.2 (60.2) 255.2 (32.4)

30 334.8 (127.9) 301.9 (87.9) 269.0 (57.9)

60 333.0 (122.1) 305.9 (81.3) 278.9 (52.7)

Note: the figures out of the brackets are the expectations while those

in the brackets are the standard deviations.


Automa2ng Ads Futures/Op2on Contracts

•  Need Ad’s Futures Contract and Risk-‐reduc5on Capabili5es –  Technologies are constrained mainly to “spots” markets, i.e., any transacJon where delivery takes place right away (in Real-‐Jme AdverJsing and Sponsored Search)

– No principled technologies to support efficient forward pricing &risk management mechanisms

•  If we got Futures Market or provide Op2on Contracts, adverJsers could lock in the campaign cost and Publishers could lock in a profit in the future


Adve

rtise

r Demand Side

Platform (DSP)

RTB / Spot Exchange

Supply Side

Platform (SSP) Pu

blish

er

3rd party data providers, ad serving, ad agency, ad

networks, campaign analytics

-10% to -30%

Futures Exchange (Programma2c Guarantee)

Futures Exchange

50

Acknowledgements

•  Thanks to my PhD students Weinan Zhang, Shuai Yuan, Marc Sloan, Xiaoxue Zhao


For more informa2on, please refer to 1.  Wang, Jun, and Jianhan Zhu. "Porqolio theory of informaJon retrieval."

SIGIR, 2009. 2.  Jin, Xiaoran, Marc Sloan, and Jun Wang. "InteracJve exploratory search

for mulJ page search results." WWW, 2013. 3.  Zhang, Weinan, et al. "To personalize or not: a risk management

perspecJve." Proceedings of the 7th ACM conference on Recommender systems. ACM, 2013.

4.  Gorla, Jagadeesh, et al. "ProbabilisJc group recommendaJon via informaJon matching." WWW, 2013.

5.  Shuai Yuan, Jun Wang, Real-‐Jme Bidding for Online AdverJsing: Measurement and Analysis, AdKDD’13 hgp://arxiv-‐web3.library.cornell.edu/abs/1306.6542

6.  Weinan Zhang, Shuai Yuan, Jun Wang, OpJmal Real-‐Time Bidding for Display AdverJsing, KDD’14

7.  Shuai Yuan, Jun Wang, Bowei Chen, An Empirical Study of Reserve Price OpJmisaJon in Real-‐Time Bidding

8.  Bowei Chen, Jun Wang, Ingemar Cox, and Mohan Kankanhalli, MulJ-‐Keyword MulJ-‐Click OpJon Contracts for Sponsored Search AdverJsing, under submission, 2013 hgp://arxiv.org/abs/1307.4980

Thanks for your aFen2on

53

Technology

On Search, Personalisation and Real-time Advertising