Contours and Convergence

Past, Present and Future (interactive) IR Research Practice

Diane Kelly University of North Carolina March 26, 2013

¤  “At the same time, even if the current retrieval environment is new, it is important to understand the message of past research and not to invent old wheels, especially square ones.”

Spärck Jones, K. & Willet, P. (1997). Readings in Information Retrieval,

Morgan Kaufmann Publishers.

Contours and Convergence

¤  con � tour ¤  an outline, esp. one representing or bounding the shape or

form of something.

¤  con �ver � gence ¤  (of several people or things) coming together from different

directions so as eventually to meet;

¤  the tendency of unrelated animals and plants to evolve superficially similar characteristics under similar environmental conditions.

‘User’ Studies in IR

Script A rhetorical device that uses sample interactions among actors.

Professor: So, what sort of project would you like to work on?

Student: Search interfaces.

Student: Well, you know how Search Engine X has blue hyperlinks? What if we used green hyperlinks instead?

Student: Yeah. Do you know how Search Engine X puts 10 results on each SERP?

Student: What if we put 12 instead?

Student: Yes. Do you know how Search Engine X ….

Professor:

Professor: That sounds good, what specifically would you like to do?

Professor: Umm, okay. That’s interesting. Do you have any more ideas?

Professor: Okay … any more ideas?

Professor: Yes?

Contexts of Practice

1990s Present

Contexts of Practice

¤  “The second unfortunate consequence of the way we worked in the sixties and seventies was that while the research community was struggling to satisfy itself in the laboratory, the operational world could not wait, and passed it by.

¤  The research experiments were so small, the theory was so impenetrable, and the results it gave were at best so marginal in degree and locus, that they all seemed irrelevant.” (emphasis added by me)

Spärck Jones, K. (1988). A look back and a look forward. Proceedings SIGIR, 13-29.

1960s and 1970s Origins of “User” Studies

Ide, E. (1967). User interaction with an automated information retrieval system. In G. Salton (Ed.) Information Storage and Retrieval: Scientific Report No. ISR-12.

Interactive Bibliographic Search: The User/Computer Interface

¤  Workshop held in January 1971; was the first attempt to evaluate the accumulating research about interactive search systems

¤  Challenges identified for discussion at workshop included: ¤  Characteristics of searchers

¤  Conceptual models presented to searchers

¤  Role of feedback during search

¤  Role of evaluation and user feedback in the system design cycle

Walker, D.E. (1971). Interactive bibliographic search: The user/computer interface. Montvale, NJ: AFIPS Press.

Interactive Bibliographic Search: The User/Computer Interface

¤  Katter & McCarn, “Development appears to occur in three roughly sequential phases: ¤  The Confidence Phase

¤  The Insight Phase

¤  The Incorporation Phase”

¤  Summit, “The two aspects of the system that most impressed users were its speed and its ‘horizon-widening’ effect; i.e., the way it suggested other relevant areas of information or different approaches to the information originally sought.”

Walker, D.E. (1971). Interactive bibliographic search: The user/computer interface. Montvale, NJ: AFIPS Press.

Oddy, R. N. (1977). Information retrieval through man-machine dialogue. Journal of Documentation 33(1), 1-14.

1980s User Models and Artificial Intelligence, Part 1

SIGIR 1983

SIGIR 1983 SIGIR 1984

IP&M 1981

Allen, R. B. (1990). User models: Theory, method, and practice. International Journal of Man-Machine Studies, 32, 511-543.

Rich, E. (1983). Users are individuals: Individualizing user models. International Journal of Human-Computer Studies, 51, 323-338.

“While the term ‘user model’ emphasizes the information about the person, it is obvious that a great deal of situational, task, or environmental information may be encoded in the model.”

Croft, W. B., & Thompson, R. H. (1987). I3R: A new approach to the design of document retrieval systems. Journal of the American Society for Information Science, 38, 389-404.

Egan, D. E., et al. (1989). Formative design-evaluation of SuperBook. ACM Transactions on Information Systems, 7(1), 30-57.

Bring on the User Studies!

¤  “More importantly, I felt that the required next step in this line of work was to carry out real, rather than simulated, interactive searching, to investigate the behaviour of relevance weighting under the constraints imposed by real users, who might not be willing to look at enough documents to provide useful feedback information.”

Spärck Jones, K. (1988). A look back and a look forward. Proceedings SIGIR, 13-29.

1990s Tools for Users (This is the best decade so pay attention!)

Hearst, M. A. (1995). TileBars: Visualization of term distribution in full text information access. Proceedings of CHI ‘95, 59-66.

Veerasamy, A. & Belkin, N. J. (1996). Evaluation of a tool for visualization of information retrieval results. Proceedings of SIGIR ‘96, 85-92.

Hearst, M. A., & Pedersen, J. O. (1996). Reexamining the cluster hypothesis: Scatter/Gather on retrieval results. Proceedings of SIGIR ‘96, 76-84.

Hearst, M. A. & Karadi, C. (1997). Cat-a-Cone: An interactive interface for specifying searches and viewing retrieval results using a large category hierarchy. Proceedings of SIGIR ‘97.

Golovchinsky, G. & Chignell, M. H. (1997). The newspaper as an information exploration metaphor. Information Processing & Management, 33(5), 663-683.

Robertson, et al. (1998) Data Mountain: Using spatial memory for document management. Proceedings of UIST ‘98, 153-162.

Belkin, N. J., et al. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434.

2000s User Models and Artificial Intelligence, Part 2

Trends

¤  Explicit feedback and anything else involving user effort – whether physical or cognitive – was declared ‘bad’

¤  Enter: studies about implicit feedback, large-scale log analysis and machine learning

Downey, D., Dumais, S. and Horvitz, E. (2007). Models of searching and browsing: Languages, studies and application. In Proceedings of IJCAI 2007, 2740--‐2747.

Large Scale Log Analysis

Clicks Queries …

model Treatment

A B

evaluate

Benefits

¤  Preeminence of behavioral and ‘user’ research (not just interfaces) and modeling

¤  Inclusion of more diverse populations

¤  Use of more varied statistical approaches to analysis

¤  Increase in interactive IR research and user studies

Limitations

Clicks Queries …

model Treatment

A B

evaluate

Clicks Queries …

model Treatment

A B model

Ide, E. (1967). User interaction with an automated information retrieval system. In G. Salton (Ed.) Information Storage and Retrieval: Scientific Report No. ISR-12.

Convergence

¤  “I was at the same time somewhat discouraged by the general lack of snap, crackle and pop evident in IR research by the end of the seventies, which did not offer stimulating new lines of work.”

Spärck Jones, K. (1988). A look back and a look forward. Proceedings SIGIR, 13-29.

Convergence

¤  con �ver � gence ¤  (of several people or things) come together from different

directions so as eventually to meet;

¤  the tendency of unrelated animals and plants to evolve superficially similar characteristics under similar environmental conditions.

Future Contours

Future Contours

¤  Expect more from people (i.e., stop treating them like lazy idiots!) ¤  Create more extensive and integrated search environments

with more tools.

¤  Revisit the issues of transparency and user control.

¤  Stop asking people if they are satisfied. ¤  Develop new methods and tools for collecting data about

interactions.

¤  Develop new measures to characterize user experience.

¤  Conduct studies to better understand method variance.

Future Contours

¤  Develop formal languages for modeling search behavior and information use ¤  Task and session models

¤  Search segmentation and modularization -> measurement

¤  Explore (create? Invent?) statistical techniques for modeling behavior, interactions and user experience

Future Contours

¤  Research Practice ¤  Slow Down!

¤  Exhausting pace leads to

¤  Flat literature base and disconnected findings

¤  Superficial or incomplete findings

¤  Mistakes

¤  Read more, experiment less.

Final Thoughts

¤  “I felt, however, somewhat flattened by the continuous experimental grind in which we had been engaged.”

¤  “But we need to stake out the ground much better than we have done so far, to be able to drive useful roads through the enormous jungle we are currently just viewing on days trips from the outside.”

Spärck Jones, K. (1988). A look back and a look forward. Proceedings SIGIR, 13-29.

Thank You.