A Research Plan to Study Impact of a Collaborative Web Search Tool on Novice's Query

BehaviorMareh Al-Sammarraie

M.S. Student

Karthikeyan UmapathyAssociate Professor,

School of Computing, University of North FloridaJacksonville, FL

http://karthikeyan.umapathy.com/

International Conference on Collaboration Technologies and Systems (CTS 2016)

Orlando, FL November 3, 2016

Work-in-Progress, Session C16

2

Collaborative Information Seeking■ Collaborative information seeking (CIS) involves people

seeking information collaboratively for a common goal, where – the information seeking is intentional, – the collaboration explicitly defined, and – the goal mutually beneficial.

■ CIS focuses on how groups of people– understand search tasks,– formulate queries,– determine results relevance,– share results,– make sense of the information, and – use information collaboratively.

3

Related fields of CIS

Information Seeking Collaboration

Collaboration

Information Seeking

(CIS)

Information

Retrieval

4

Collaborative Information Seeking (CIS) Examples■ Collaboration is a useful and often necessary component of

complex projects, and the settings for CIS are diverse. ■ In CIS context, people come together with intention, looking for

and sharing information, and making sense out of shared findings to attain their goals.

■ Business/ Professional contexts– a team of engineers working together to create an

infrastructure design, – researchers coauthoring a scholarly article, – a group of medical professionals diagnosing a patient, and – a team of programmers solving software development tasks.

■ Social/Family contexts– a family planning a two-week vacation, and – a couple organizing their wedding.

5

Studying CIS Practices■ People collaborating on information-seeking tasks work in distinct

ways, they should be studied and supported separately.

■ Predominant literature focuses on and considers information retrieval as an individual activity.

■ While there has been a growing interest in understanding and supporting CIS practices, we still lack clear understanding of people’s CIS behavior.

■ CIS activities are supported by collaborative search tools. Thus, investigating and gaining better understanding of these activities is important for designing and developing effective collaborative web search tools.

6

CIS Tools■ Collaborative search tools are expected to provide

– awareness features; that ■ archive group member query histories, ■ selected results, and ■ comments,

– division of labor features; which include ■ chat systems, ■ ability to divide search tasks, and ■ selecting search results based on a group member's action.

■ Currently available, known and/or researched, CIS tools are:– SearchTeam (http://searchteam.com/)– Coagmento (http://www.coagmento.org/)

– SearchTogether (http://research.microsoft.com/en-us/um/redmond/projects/searchtogether/)

7

SearchTeam

8

Studying CIS Tools■ CIS tools have minimal to none adoption rates.■ Despite availability of CIS tools, people use general purpose

search tools like Google along with communication technologies (email, texting, IM, and social media) that are part of their everyday routine.

■ Conducting comparative study between CIS tool and Google integrated platform for CIS activities are important, as it could provide insights into CIS features needed for designing and developing effective collaborative web search tools.

9

Motivating Scenario■ Motivation for this research study is raised from researcher’s

personal experience as well as from interactions with local software companies.

■ When an intern software developer is recruited by a software company, an intern is assigned to an experienced software developer.

■ Intern is expected to learn how to troubleshoot software development problems by working along with an expert.

■ Interactions between the interns and the experts are essentially a CIS process.

■ In this context, the question of effectiveness of using collaborative search systems like SearchTeam as opposed to integrated Google platform was posed to researchers.

■ The main research objective for the empirical study is to assess effectiveness of SearchTeam tool in the context of novice-expert pairs troubleshooting software development problems.

10

Research Context■ Web information retrieval process can be divided into three stages:

finding documents, query formulations, and determining document relevance.

■ In this research study, we focus on the second stage, query formulations as it requires user action.

■ Research-in-progress paper proposes a research plan to examine: 1. The effect of users’ domain expertise on users’ query behavior

in terms of basic query features, query reformulation behaviors, and query performance in the context of collaborative web search; and

2. The impact of using tools designed for collaborative information seeking (CIS) process in aiding novices to learn effective querying strategies from domain experts through collaborative problem solving.

■ We are interested in gaining an in-depth understanding of the impact of pairing novices with experts in the collaborative search process.

11

Research Questions and HypothesesResearch Questions:■ Are there any differences among collaborators who are using a traditional

search engine (Google) as opposed to using a collaborative Web search tool (SearchTeam) from perspectives of query features including query vocabulary richness, number of queries, classes/patterns of query reformulations, and query performance?

We hypothesize that:■ Non-experts collaboratively solving problems with an expert using SearchTeam

would have better query behaviors than solving with other non-experts.■ Collaborative teams that use SearchTeam would need fewer numbers of queries

with more accurate terms for the difficult search tasks than teams that use Google.

■ Collaborative teams that use SearchTeam would tend to use new and specialized query reformulation more often than teams that use Google.

■ Collaborative teams that use SearchTeam would require less time to search for both simple and complex tasks with higher query performance rates than teams that use Google.

■ Domain experts collaborating with non-experts, using SearchTeam, will immensely increase the query success rates for novice users, and help them improve their querying strategies over the course of time.

12

Research Plan

Collaborative Team

Collaborative Search Systems

SearchTeam GoogleNovice-Expert Team Group A Group BNovice-Novice Team Group C Group D

■ Research Method: Controlled Lab Experiments. ■ Experimental Design: 2x2 between subjects design.

■ Novices participants will be junior or senior computing students. Plan to recruit 40 students.

■ Experts will be software developers with more than 5 years of experience. Plan to recruit 20 experts.

■ Participants will be divided into teams of two. Each team will be randomly assigned to use either SearchTeam or Google to complete search tasks collaboratively.

■ We plan to have 10 teams for each experimental group.

* Google Integrated Platform: Google web search, Google Docs, and Hangout.

H1 – (A vs. C) vs. (B vs. D)H2, H3, &H4 – A+C vs. B+DH5 – A vs. B

13

Search Task: ASP MVC ProblemsProblem Context:■ Bookstore Management App is a web application that automates all operations of

an online book store. Using this application, users will be able to search the online book catalog and place an order for a book. The store manager will be able to manage the book catalog, and review and fulfill book orders.

Instructions given to participants:■ The paired teams will be expected to work together remotely but at the same time

to identify relevant solutions for the problems. ■ Team members will be placed in different lab rooms to simulate remotely-located

collaboration to ensure participants use chat for communication and read each other’s search histories and saved information in the team’s workspace.

■ The paired teams will be expected to search for relevant solutions using appropriate query terms and select a result item as the solution for the given problem.

■ The teams will be encouraged to review each other’s query terms and engage in an iterative search process to reach consensus on which result item describes the most ideal solution.

■ While participants will receive source codes, they are not expected to implement those codes and solve the problems.

14

Simple Search Task: Web Form Data Entry Validation■ Task 1: Client-side Form Validation

– Identify an appropriate web search result that describes the process for adding client-side form validation for text box fields.

15

Task 2: Server-side Form Validation■ Entries for ‘New Release Coming Soon’ and ‘New Release: Last

30 Days’ fields need to be validated at the server-side.

16

Complex Search Task: URL Routing■ Task 3: Add New URL routing for Index Action

– Direct Get call from three additional URLs to trigger the ‘Index()’ action in the ‘Books’ controller.

■ Task 4: Restrict access to default URL– Restrict access to default URL to only authenticated users.

17

Data Collection and Measures■ We will collect data from transaction logs such as users’ queries (search

terms), search results selected as a solution, and users’ actions, such as procedure followed to come up with relevant results collaboratively.

■ Data gathered will be analyzed using independent sample t-Test to compare means of experimental groups.

■ We will analyze the data in three dimensions: – Basic query features

■ Total Number of Queries (TNQ) = summation of team member queries■ Vocabulary Richness (QVR) = No. of unique queries / Total No. of queries

– Query performance = Queries with items saved / Total no. of queries– Query reformulation patterns

Type DefinitionNew If Q1 is the first issued query , and does not share any common

terms with Q1+1Generalization

Q1 and Q1+1 share common terms , and Q1+1 contains fewer terms than Q1

Specialization

Q1 and Q1+1 share common terms , and Q1+1 contains more terms than Q1

Reconstruction

Q1 and Q1+1 share common terms , and Q1+1 has the same length as Q1

18

Conclusion■ Learning how users with different levels of expertise reformulate their queries

in collaborative searches can help search systems to improve query suggestion feature for group users.

■ This paper contributes by presenting a research plan to investigate influence of CIS tools in CIS contexts.

■ Anticipated findings– Successful query rates in collaborative search teams, where at least one

of the team members has sufficient knowledge about the domain are expected to be higher when collaborative web search tool is used.

– Having domain experts in collaborative search team working with novices will help novices pick up new terms, querying strategies, and tricks to identify reliable sources.

■ Future studies– Focus on factors that impact effective usage of CIS tools and improve CIS

process efficiencies.– Would usage of CIS tools help in transfer of domain knowledge from

experts to novices?■ Findings from empirical investigations can help designers determine

importance of collaborative search system features and CIS process activities.

19

Thank You!