Modeling)Rich)Interac1ons)in)Session) …infosense.cs.georgetown.edu/publication/presentation.pdf · OurSubmissions 16 GUS14RUN1! GUS14RUN2! GUS14RUN3! RL1! AdNhoc!Retrieval!Model!

Modeling Rich Interac1ons in Session Search – Georgetown University at TREC

2014 Session Track

Jiyun Luo, Xuchu Dong and Grace Hui Yang Department of Computer Science

Georgetown University

Introduc:on •  Session search

–  Document retrieval for an en:re search session. •  TREC Session Track provides log data which records

–  A sequence of query changes q1,q2…qn-‐1,qn –  The ranked list for each past query –  Document clicked informa:on and dwell :me.

•  TREC 2014 Session Track: –  RL1 using the last query of a session –  RL2 using any informa:on in current session –  RL3 using informa:on from other sessions

•  We use: –  ClueWeb12 Category A as our corpus

2

Outline •  Introduc:on •  Methods and Approaches – Ad-‐hoc Retrieval Model (Ad-‐hoc) – Query Change Retrieval Model (QCM) – Weighted QCM – User-‐Click Model –  Clustering –  Session Performance Predic:on and Replacement

•  Submissions •  Evalua:on Result •  Conclusion

3

Ad-‐hoc Retrieval Model (Ad-‐hoc)

•  Mul:nomial Language Modeling + Dirichlet Smoothing.

•  Term weight P(t|d) as:

•  μ is the Dirichlet smoothing parameter, and is set = 5000.

4

Query Change Retrieve Model (QCM) •  Idea: Query Change is an important form of user feedback

–  Dongyi Guan, Sicong Zhang, and Hui Yang. 2013. U:lizing query change for session search. (SIGIR '13).

•  Defining query change as the syntac:c edi:ng changes between two adjacent queries:

•  Added term ; Removed term ; Theme term

1−−=Δ iii qqq

iqΔ

5

Session Queries Query Change Qtheme

session 52

Q1 = hydropower efficiency +Δq2 = environment hydropower

Q2 = hydropower environment -‐Δq2 = efficiency

Q3 = hydropower damage +Δq3 = damage hydropower

-‐Δq3 = environment

iqΔ+ iqΔ− themeq

Table 1 A example of Query Change

Query Change Retrieve Model (QCM)

•  The relevance score between one query qi and a document d is calculated by:

6

Score(qi,d) = logP(qi | d)+αWTheme −βWAdd,In +εW Add,Out−δWRemove

Current reward/ relevance score

Increase weights for theme terms

Decrease weights for old added

terms

Decrease weights for removed terms

Increase weights for novel added

terms

Query Change Retrieve Model (QCM)

•  The relevance score between one query qi and a document d is calculated by:

•  The QCM model combines all queries in a session with a discount factor Υ:

7

Score(qi,d) = logP(qi | d)+αWTheme −βWAdd,In +εW Add,Out−δWRemove

Scoreqcm (q1..n,d) = γ n−iScore(qi,d)i=1

n

∑

Weighted QCM •  Weighted QCM combines queries based on query quality which is indicated by user click –  Strong SAT-‐Click a clicked document with dwelled :me >= 30 seconds –  Weak SAT-‐Click a clicked document with dwell :me >= 10 seconds and<

30 seconds

8

Weighted QCM •  Weighted QCM combines queries based on query quality which is indicated by user click –  Strong SAT-‐Click a clicked document with dwelled :me >= 30 seconds –  Weak SAT-‐Click a clicked document with dwell :me >= 10 seconds and<

30 seconds

9

!"#$%!"#$ !!..!,! = !"#$% (!! ,!)!!∈!!""#

+ !! !"#$% (!! ,!)!!∈!!"#

!

The good query set: Queries bringing at least one SAT-‐Click + the current query

The bad query set: Queries bringing no SAT-‐Click

User-‐Click Model •  We boost a document’s ranking score, if it is SAT-‐Clicked by users –  Session Level User-‐Click Model for RL2

10

score from QCM model

boost from Session level User-‐Click model

Ψ points for a Strong SAT-‐Click, θ points for a Weak SAT-‐Click, sum up for the whole session

normaliza1on to (0,1)

User-‐Click Model –  Topic Level User-‐Click Model for RL3

•  A session cluster is a set of sessions that sharing similar search topics

11

boost from Topic level User-‐Click model

similar to session level User-‐Click model, however calcula:on is done for the whole session cluster

Clustering •  Topic ID is not obtainable in real search prac:ce.

–  cluster sessions by comparing queries’ similarity

Ø Convert all queries in one session to a term vector Ø Assign idf value as weight to each dimension Ø Cluster sessions based on the Euclidean distance of these vectors

•  We use K-‐means clustering algorithm and set K = 60

12

Session Performance Predic1on and Replacement

•  For sessions that share similar search topics –  predict their performance –  replace bad sessions’ results with good sessions’

•  Predict session performance –  Extract several features (n) from the sessions –  Rank sessions by formula:

13

!"#$%! ! = 1#!!"!!"!!#$%!!!"#$!%&$'(!!! = TRUE

!!!..!∗ !(!!)!

Session Performance Predic1on and Replacement

14

Table&2&Features&Extracted&for&each&Session&

Feature Definition F1 Search intent is comparison F2 No user-click in session s F3 !!"#$$ ≤5s. !!"#$$ !!"!!ℎ!!!"#!!"!!"#$$!time in a session. F4 # of unique terms in session s≥20. F5

!!"#$$_!"#_!"#!$ <!!"#$$_!"#_!"#!$(!)

2

F6 Session s does not contain the most frequent search term in T(s). F7 # of unique terms in session s≤6 F8 #!!"!!"#!!"#!$%!!"!!"!!#$%!! < #!!"!!"#!!"#!$%!!"!!"!!#$%!!′!!∈!(!)

|!(!)|

!

•  Features Table

* T(s) means a session cluster including session s

Outline •  Introduc:on •  Methods and Approaches – Ad-‐hoc Retrieval Model (Ad-‐hoc) – Query Change Retrieval Model (QCM) – Weighted QCM – User-‐Click Model –  Clustering –  Session Performance Predic:on and Replacement

•  Submissions •  Evalua:on Result •  Conclusion

15

Our Submissions

16

GUS14RUN1 GUS14RUN2 GUS14RUN3 RL1 Ad-‐hoc Retrieval Model

RL2

•  Weighted QCM (ω=0.65)

•  Session Level User-‐Click Model

•  Weighted QCM (ω=0.8) •  Session Level User-‐Click Model

RL3


•  Topic Level User-‐Click Model


•  Topic Level User-‐Click Model

•  Weighted QCM (ω=0.8) •  Topic Level User-‐Click

Model using topic ids •  Session Performance

Predic:on and Replacement

•  RL3 in RUN1 and RUN2 using session clusters based on query similarity •  RL3 in RUN3 using session cluster based on topic id

•  Why? similar queries leads to similar retrieval list in our system. Not useful when apply session replacement strategy

Evalua1on Results

17

GUS14RUN1 GUS14RUN2 GUS14RUN3 Max Med

nDCG@10 P@10 nDCG@10 P@10 nDCG@10 P@10 nDCG@10 P@10 nDCG@10 P@10

RL1 0.2053 0.378 0.2053 0.378 0.2053 0.378 0.3890 0.629 0.1549 0.348

RL2 0.2458 0.426 0.2482 0.427 0.2482 0.427 0.4865 0.712 0.1626 0.372

RL3 0.2443 0.423 0.2458 0.424 0.2580 0.429 0.5111 0.744 0.1790 0.404

Ø  2nd rank in task RL1, 1st rank in task RL2 and RL3 •  Adjus:ng term weight based on query change is effec:ve •  Combining queries in a session is useful for Session Track •  User-‐Click is effec:ve to predicate relevance

Ø  A small performance drop from RL2 to RL3 in RUN1 and RUN2 •  cluster sessions based on query similarity may work, however need more work to

refine it Ø  A small increase from RL2 to RL3 in RUN3

•  For sessions sharing same search topics, replacing poor sessions’ results using good sessions’ is prac:cal.

•  Achieve 20.9% increase from RL1 to RL2 by u:lizing – query change feedback – user click feedback

•  Achieve 4% increase from RL2 to RL3 by – Topic level User-‐Click Model – Session performance predic:on and replacement

Conclusion

Thanks!

19

Jiyun Luo, Xuchu Dong and Grace Hui Yang

Department of Computer Science Georgetown University

Documents

Modeling)Rich)Interac1ons)in)Session) …infosense.cs.georgetown.edu/publication/presentation.pdf · OurSubmissions 16 GUS14RUN1! GUS14RUN2! GUS14RUN3! RL1! AdNhoc!Retrieval!Model!