Click Chain Model in Web Search Fan Guo Carnegie Mellon University 11/29/2014WWW'09, Madrid, Spain

Click Chain Model in Web Search

Fan GuoCarnegie Mellon University

104/10/23 WWW'09, Madrid, Spain

Chao LiuMSR, ISRC-Redmond

Yi-Min WangMSR, ISRC-Redmond

MSR, CambridgeMike Taylor

MSR, Search LabAnitha Kannan

MSR, CambridgeTom Minka

Carnegie Mellon UniversityChristos Faloutsos

Joint Work With…

04/10/23 WWW'09, Madrid, Spain 4

Click Logs

• Auto-generated data keeping important information about search activity.


Rank/Position URL of Document Click1 www.metalwayfestival.com 0

2 www.maquitec. com 03 www.construmat.com 04 www.hispack.com 05 www.themarket.com 06 www.cursabombers.com 07 www.setegibernau.com 08 www2009.org 19 www.solardecathlon.upe.es 0

10 www.nxtbook.com/nxtbooks/suny/2009spring 0

Query www 2009 Time 21 Apr 2009, 9:01:02

Problem Definition

• Given a click log data set, for each query-document pair, compute user-perceived relevance.


Rank/Position Document Idx Click1 1 02 8 03 3 04 7 05 5 06 12 07 2 08 5 19 42 0

10 20 0

Query www 2009

Session Index 103

…

Document Idx Relevance1 ?

2 ?

3 ?

4 ?

5 ?

6 ?

7 ?

8 ?

9 ?

…

Impression Data

Click Data

Relevance Representation


Excellent

Good

Fair

Bad

0 1

Click Chain Model

0.75

Previous Click ModelsHuman Judge

Integration

Applications

• Automated Ranking Alterations

• Search Engine Performance Metric

• Calibrate Human Judgment

• Related Application in Sponsored Search


Roadmap

• Motivation and Problem Definition• Click Model Basics• CCM and Algorithms• Experimental Evaluation• Related Work and Conclusion



Eye-Tracking User Study


Fixation Heat Map

• Overall: Fixation is biased towards higher ranks, so do the clicks.

• For each position:fixation/clicks are context dependent.


Normal Impression

Reversed Impression

Problem Definition (Recap)

• Given a click log data set, for each query-document pair, compute user-perceived relevance and the solution should be– Aware of the position bias and context

dependency– Scalable to Terabyte data– Incremental to stay updated

04/10/23 13WWW'09, Madrid, Spain

Examination Hypothesis

• User behavior abstraction:Fixation → binary examination variableClick → binary click variable

• A document must be examined before being clicked.



• For each position, P(Click=1) = P(Examination=1) * Relevance Relevance = P(Click=1|Examination=1)

• The position bias is reflected in the derivation of P(Examination).


• User scans through documents and make decisions in strict linear order.

• The decision process: E1, C1, E2, C2,…

• Essential part of click model:– What is the probability of “See Next Doc”?

Cascade Hypothesis


Roadmap



The Context• Top-10 organic search results only.

• Query sessions are independent.• Semantic info are not used.


Suggestions

Ads

Other Elements

User Behavior Description


Examine the Document

Click?

See Next Doc?

DoneNo

Yes

Yes

No

Yes

iR

1 iRSee Next

Doc?

DoneNo

2 31 i iR R

C4C3C2C1

Click Chain Model

20

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

Why Bayesian?

• Modeling Benefit:– A principled way of smoothing the relevance

estimates;– Offers more flexibility such as computing P(Ri>Rj).

• Computational Benefit:– Avoid iterative optimization procedure in

maximum-likelihood estimation


log 1 j iR

Relevance Inference

• Given a query, and all its click data compute the posterior for each possible j.

• Let then focus on click probability for a particular

session, and look at different cases 04/10/23 WWW'09, Madrid, Spain 22

1,..., NC CC

|jp R C

1

| |N

nj j j

n

p R p R P C R

C

C4C3C2C1

Click Chain Model

23

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5


Cascade Hypothesis

C4C3C2C1

24

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

1 1Case I: | 1 P C R R

0 1 0 1

C4C3C2C1

25

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

2 2 3 2 2| 1 1Case I /I: P C R R R

0 1 0 1

C4C3C2C1

26

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

2 33 3 3

1 2

|Case III: 12

P C R R R

0 1 0 1

C4C3C2C1

27

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

4 4 4Case IV: | 1 P C R R

0 1 0 1

C4C3C2C1

28

R1

E1 E2

R2 R3 R4

E3 E4

…

…

…


C5

R5

E5

5 5 5Case IV: | 1 P C R R

0 1 0 1

Putting them together


0| 1m

jm

K

j jK

mp R R R C

1Case I: K

2 0Case II: , K K

3 0Case III: , K K

3Case IV: ,

where is the last clicked position.

j lK

l

Summary of the Algorithm

• Initializing (2*10+2) counts for each pair;• Go through the click log once and update the

counts;• Compute parameter values and get β values;• Ready to output results (using numerical

integration if necessary).


Sanity Check

• The algorithm should be– Aware of the position bias and context

dependency

– Scalable to Terabyte data Single Pass, Linear

– Incremental to stay updated Update counts

04/10/23 31WWW'09, Madrid, Spain

Roadmap



Data Set

• Collected in 2 weeks in July 2008.• Preprocessing:

– Discard no-click sessions for fair comparison.– 178 most frequent queries removed.

• Split to training/test sets according to time stamps.


Data Set

• After preprocessing:– 110,630 distinct queries;– 4.8M/4.0M query sessions in the training/test set.


Metric

• Efficiency:– Computational Time

• Effectiveness: – With known document identities in the test set,– Using the relevance and parameter learned on the

training set, – To do Click Prediction.


(resort to indirect measure)

Competitors

• UBM: User Browsing Model (Dupret et al., SIGIR’08)

– More parameters– Iterative, more expensive algorithm

• DCM: Dependent Click Model (WSDM’09)

– Modeling 1+ clicks per session


Results - Time

• Environment: Unix Server, 2.8GHz cores, MATLAB R2008b.


CCM UBM DCM9.8 min 333 min 5.4 min

1.0 34 0.55

Results – Perplexity

• Perplexity: quality of click prediction for each position individually.


/ /entropyperplexity 2 1/ 1/H TN N N N

H Tp p

Random Guess (pH=0.5): 2.00Best Guess (pH=0.8): 1.65Ground Truth (Cheating): 1.00



Worse

Better


• Average Perplexity over top 10 positions.


Model CCM UBM DCMPerplexity 1.1479 1.1577 1.1590Equiv. PH 0.0309 0.0334 0.0337

Improv. 7.5% 8.3%

Results – Log Likelihood

• Log-likelihood: log of the chance to recover the entire click vector out of 210 possibilities.


Model CCM UBM DCMLL -1.171 -1.264 -1.302

Likelihood 0.3100 0.2719 0.2826Improv. 9.7% 14%

Results – Log Likelihood


Better

Worse

Roadmap



Related Work

• User behavior study and hypothesis– Eye-tracking Study (Joachims et al., KDD’05, ACM TOIS)

– Examination Hypothesis (Richardson et al., WWW’07)

– Cascade Hypothesis (Craswell et al., WSDM’08)

• Other click models– Logistic Regression (Dupret et al., SIGIR’08)

– Dynamic Bayesian Network (Chapelle et al., WWW’09)

– Bayesian Browsing Model (KDD’09, To appear)


Conclusion

• Click Chain Model– A probabilistic approach to interpret clicks.– A Bayesian approach to model relevance.– Both scalable and incremental.

• Future Directions– Validation/Bucket Test.– Pairwise comparison– More on context dependency


Thank you :-)


Abstract/Document Relevance

• Relevance of Abstract: – Conditional probability of click as defined by

examination hypothesis

• Relevance of Document:– Determines the probability of “See Next Doc”– A binary random variable (integrated out under CCM)


~ ( ), 1| 1abstract i i i abstractr p R P C E r

11 2 3

~ ( )

1| 1, 1 document document

document abstract

r ri i i

r Bernoulli r

P E E C

Alt. User Behavior Description


Examine the Document

Click?

Relevant?

Yes

Yes

Yes

No

No

See Next Doc?

See Next Doc?

See Next Doc?ir

~ ( )i ir p R Yes

2

Yes

3

Results – Perplexity (by Freq)


Worse

Better

Examination/Click Distribution


Predicting First/Last Clicks

• Root-Mean-Square error in predicting the first/last clicked position for the test data.

• Two approaches (bias/variance tradeoff):– EXPectation: using the expected value (bias)– SIMulation: drawing sample from the model

(variance)


First Clicked Position


Last Clicked Position


A Quick Example

• Here we are interested in R3

54

-1 -0.63 0.83 -0.33

0 1 2 3

0

4 5 6

0 0

..

-0.11 -0

0 0 0

.

.

04

m

m

m

K

0


0| 1m

jm

K

j jK

mp R R R C

A Quick Example


55

-1 -0.63 0.83 -0.33

0 1 2 3

4 5 6 .

0 0 1 1

-0.11

0

-0.0

0

4

..

m

m

m

K

0


C4C3C2C1

A Quick Example


56

-1 -0.63 0.83 -0.33

0 1 2 3

1

4 5 6

0

..

1

-0.11

.

0

0

-0.04

1

m

m

m

K

0


C4C3C2C1

C4C3C2C1

A Quick Example


57

-1 -0.63 0.83 -0.33

0 1 2 3

1

4 5 6

1 0

..

-0.11 -0

1 0

.

.

1

04

m

m

m

K

0


C4C3C2C1

C4C3C2C1

C4C3C2C1

A Quick Example


58

-1 -0.63 0.83 -0.33

0 1 2 3

1

4 5 6

1 0

..

-0.11 -0

1 0 1

.

.

04

m

m

m

K

0


3 3 3 3 3| 1 1 0.83 1 0.11p R R R R R C

Mean(R3) = 0.52Std(R3) = 0.22

Documents

Click Chain Model in Web Search Fan Guo Carnegie Mellon University 11/29/2014WWW'09, Madrid, Spain