Cummings Research Proposal

Traditional and Next-Generation OPACs: A Usability Study

Darcy F. Cummings

San Jose State University

May 11, 2014

Traditional and Next-Generation OPACS: A Usability Study

Introduction

The ways in which library users search for information, the expectations of end users for

information search tools and the conditions that facilitate the ease of information seeking have

changed considerably since the beginning of online public access catalogs (OPACs); this is

primarily due to the considerable use of web based search engines such as Google, Amazon,

EBay and Yahoo, to name a few. Users are actively searching for information via the web

before any formal academic or library search skills training occurs in their lives. Due to this, the

concept of OPACs and their formatting needs to change to remain relevant and useful. Mi and

Weng (2008) stated: “The information-seeking behavior of academic library users has drastically

changed in recent years. Internet search engines have become the preferred tool over the library

online public access catalog (OPAC) for finding information” (p. 5).

Traditional OPACs are often perceived as antiquated and more useful for skilled

searchers and/or librarians rather than average or novice users. In response, libraries have started

to revamp their traditional OPACs and make them feel more like a general web-based search

engine such as Google. These new OPACs are referred to as next-generation OPACs and are

still being studied and analyzed to ensure that they are meeting user needs. Experiments with

next-generation OPACs have been mostly in academic libraries. Some design changes include

simplified designs with uncluttered search pages and an unmistakable search box featured

prominently on the home page and an advanced search option directly below it via hyperlink.

The search limiters are located to the side of the search results and use simplified terminology as

well as suggestions for narrowing the search such as related subjects. Next-generation OPACs

are designed to be easier for the novice user to search and to refine searches to achieve relevant

search outcomes. Users need not be versed in Boolean search terms to use them; also, one of the

2


most substantial differences between traditional and next-generation OPACs is that in the former

users are encouraged to limit their search before the search results are returned and in the latter

users are encouraged to limit their searches after the search results are returned.

Two predominant factors when assessing the advantages of either a traditional or a next-

generation OPAC are user-friendliness and search outcomes. User-friendliness is the user’s

perception of how easy and straightforward it is to understand and use the search interface and

search features. Examples are things such as fluidly navigating from page to page or through

search results, understanding the terms used, being able to find necessary search features and

being able to narrow or broaden search results using search features. Search outcomes deal with

how effectively the user was able to find results that were relevant and how satisfied they are

with the search results. An example that illustrates both of these would be a search looking for

information about an author, rather than a search for information by that author. User-

friendliness, in this example, would consist of being able to easily limit search results to an

author as subject and search outcomes would consist of relevant results about the author as

subject.

There has been considerable research done on this issue, both in evaluating how

traditional OPACs fail to meet user needs and expectations as well as the shortcomings of next-

generation OPACs that were designed to remedy the current generation’s issues with traditional

OPACs. Many next-generation OPACs are still in the process of being analyzed and the

literature regarding those is insufficiently substantial: most have only been through one or two

research studies. What is interesting, however, is to note the flaws found with traditional

OPACs, how they have been treated in next-generation OPACs as well as what is still flawed.

3


This research proposal seeks to build on what is already known about how students use

OPACs and what makes them easy or difficult to use and to explore in greater detail exactly

what the differences in perceived ease or difficulty of use between a traditional OPAC and a

next-generation OPAC are. By doing so, the researchers endeavors to illustrate what is still not

working with next-generation OPACs and how this can be remedied as well as contribute to the

ongoing research of next-generation OPACs. The research question, then, is: what are the

factors that impact OPACs’ perceived user-friendliness and search outcomes from undergraduate

students’ perspectives?

Literature Review

Changing User Needs

Liu and Luo (2011) stated: “To provide effective digital library services, designers and

managers need to have a good understanding of the factors that influence use and non-use of

digital libraries, user’s perception, and satisfaction” (p. 230). This brought to light the question

not how are users using OPACs, but why are users not using OPACs in their research and instead

relying upon web based research tools? The answer that many researchers have found is ease of

use and user perceptions of finding results in OPACs versus the same in finding results using the

Internet. In other words, as Johnson and Craven (2010) put it: “users’ immediate and reflective

responses regarding both the usability and perceived usefulness of system features” (p. 228).

Johnson and Craven (2010) also found key factors to measuring user needs in terms of OPAC

usability: “learn-ability, ease of use, aesthetic appearance, navigation, and terminology” (p. 234).

As the way users are accustomed to searching becomes more ingrained through continual usage

of web based search engines, users expect the same simplicity, language, maneuverability and

ease of understanding in all of their search mediums.

4


Traditional OPAC Issues

The interface design of traditional OPACs is typically one that uses Boolean operators as

the default search method. For users that typically only use web based search engines and have

not been taught to use Boolean operators there is a disconnect with this default search tool.

Dinet, Favart and Passerault (2004) found that: “the relevance of presenting explicit Boolean

operators in OPACs is undermined if users have not previously received formal training in the

use of these operators” (p. 339). The authors’ research was conducted to find whether there were

overt differences in the use of Boolean operators amongst three categories of users: beginner,

intermediate and expert; the authors concluded that there was a substantial difference in

understanding and use of Boolean operators between the three groups, with beginner users rarely

if ever using them and expert users (trained librarians) almost always using them (Dinet, et al.,

2004).

Liu and Luo (2011) found that “students like easily-mastered search engines” (p. 233).

For generations of users who have grown up using search engines such as Google and Yahoo,

the ease of use in terms of learning to use these search engines is essentially unmatched in the

academic library OPAC setting. The information found may be more reliable, useful and

pertinent, but the obstacle of learning to use the OPAC may overcome the benefits. Mi and

Weng (2009) concluded that: “Teaching users to search for structured bibliographic data is

completely opposed to the ever-popular free and open Internet search mechanism drawn from

Google-like search experience, which does not require any special training” (p. 6). As Johnson

and Craven (2010) wrote: “The Web and specifically search engines have changed user

expectations and academic librarians and web technologists are increasingly rising to the call to

‘make it like Google’” (p. 230).

5


Next-Generation OPACs

A growing number of academic libraries are capitulating to the demand that their OPAC

search interfaces become more like web based search engines. The demand of users for easier to

understand and use OPACs is behind this shift in design. Hessel and Fransen (2012) stated that:

“Many libraries have introduced next-generation catalogs to satisfy the needs and expectations of

a new generation of library searchers. These catalogs incorporate some of the features that make

the current web environment appealing…” (p. 21). One aspect of next-generation catalogs is

making the search interface look more like a basic web search engine interface, another is

limiting searches after the original search rather than prior to. An issue that multiple researchers

found was the wording of limiters or search features: they were more inline with traditional

OPAC and library language rather than current language used by the actual end users even in

next-generation OPACs (Johnson & Craven, 2010; Yang & Hofmann, 2010; Emanuel, 2011).

Methodology

Study Population

This research study will focus on San Jose State University (SJSU) undergraduate

students. As mentioned previously, these are the students who are less likely to be familiar and

comfortable using a traditional OPAC and instead are more likely to be comfortable using a

next-generation OPAC that replicates their previous search strategies using Internet search

engines. SJSU’s King Library currently offers access and searching through both a traditional

OPAC and a next-generation OPAC (referred to locally as the classic and new catalogs). The

aim of the study is not to show what type of users are more comfortable with either OPAC, but to

see what factors in each type of OPAC influence a user’s perceived comfort level and search

effectiveness.

6


Sampling Design

Using probability sampling I will take a list of undergraduate students enrolled full-time

at SJSU and send 75 requests to a random selection for participation in the study. Selected

students will be emailed a request to participate in the study (see appendix A). Of the 75

requests sent, I aim to gather 10-15 actual participants as my study population. If more than 10-

15 students agree to the study, then I will employ a systematic sampling method to narrow the

population down to 10-15.

Data Collection

Study participants will be asked to perform 10 search exercises (see appendix B) on both

the traditional OPAC and the next-generation OPAC of the King Library (see appendix C). The

search exercises are designed to have participants complete multiple searches for items in

different formats, within a specific date range, by author and/or subject, etc. The intent is to see

whether participants will take advantage of search features in order to make searching more

effective and easy and create greater relevance in their search results. Back end tracking devices

will be employed on the computers participants are using to record mouse movement, time spent

on each search exercise, search query terms, use of limiters (i.e. date ranges, author or subject,

format, etc.) and other screen activities during the search exercises. After the exercises have

been performed, participants will be asked to take a short survey (see appendix D) that asks

about their feelings during the search exercises as well as their personal feedback regarding

different elements of the two OPACs’ design and functionality. The survey will be administered

via the same computer used for the search exercises. Study participants will be allowed a 15-

minute break between performing the search exercises and partaking in the survey to gather their

thoughts and reduce levels of fatigue or other stressors related to the search exercises.

7


The researchers are employing these two different data gathering methods in order to

measure two distinct parts of OPAC usage: how the participants used the OPAC search features

and how the participants felt about their search experience. The back end tracking will show

precisely what steps participants took to find their search items and the survey will show how

they felt about those steps. The presumption being that participants who found it easy to use

search features such as limiters, would have a higher level of satisfaction and more positive

feelings about the search experience and that participants who did not use specific search

features and limiters would have more negative feelings about the search experience.

Survey question 1 serves as a general perspective on how the participant felt about their

search experience. Questions 2-6 are designed to show their perceptions of specific search

features in the two catalogs. Question 7 is exploratory in nature and is designed to see why

participants did not use limiters (for example: they didn’t know they existed, they were unsure

how and/or when to use them, or they were unsure what they meant because the terminology was

ambiguous or confusing). Questions 8, 9 & 11 are designed to gather data regarding

participants’ personal views on each of the catalogs’ search features. Question 10 is simply to

understand which catalog was preferred in general by the study participants and to ensure that

the data gathered by other survey questions is consistent with the participant’s preferred catalog.

In other words, if a participant consistently ranked the traditional catalog higher in previous

survey questions, but responded that they would prefer to use the next-generation catalog, then

the data might be erroneous and would need to be analyzed further.

Data Analysis

By using back end tracking devices the researcher will observe how much time is taken

on each task, what the initial search query is for each exercise, how many queries are performed

8


and how and when limiters are used in searches, as well as any additional data that would benefit

the study. The data collected from the back end tracking will help in understanding precisely

how students are performing searches, what features of each catalog they are using or not using

and provide average search times that may indicate whether or not the search was difficult or

easy. The researchers will also be comparing search times with the use of limiters to see if there

is an effect on the amount of time spent and the amount of limiters used or not used. A

comparison of participants’ search actions with their answers to the survey questions will be

necessary to see how students felt about search features, etc. versus how they actually used those

search features.

Of the 11 survey questions, 6 will be interval scale questions and the answers will be

analyzed to find the central tendency and mean of those answers. Questions 7-9 and question 11

are open-ended questions and are designed to draw out a more detailed evaluation from the

participants’ personal viewpoints. Open coding methods will be employed to find and analyze

the recurring concepts from the participants’ responses. These will be grouped in to categories

for further analysis. Question 10 is a close-ended question and the analysis will be through the

mean of the responses.

Data collected from the survey responses and the back end tracking will be analyzed

separately and then analyzed comparatively. In order to achieve this, students will be assigned

participant numbers that will be associated with their search session and follow up survey. This

will also help ensure the anonymity of participants. The researchers aim to find correlations

between what study participants felt was easy to use as well as helpful in finding relevant search

results and how participants used search features. By comparing participants’ use of search

features and feelings about those search features on the old and new catalogs, this study should

9


be able to determine what factors make OPACs user-friendly and result in relevant search

outcomes from the participants’ perspectives and actions.

Schedule

Tasks Estimated Date of CompletionObtain consent & secure budget September 2014

Conduct literature review & draft Research Proposal

October 2014

Assemble a research team October 2014

Compile participant request form, survey questionnaire, search exercises test

November 2014

Email participant request letters including informed consent forms

November 2014

Collect participant responses and select sample population

December-January 2014

Perform study including participant search exercises & post-exercise survey

February 2014

Analyze data including determining coding scheme

March-April 2014

Create rough draft of research findings May 2014

Present findings for review June 2014

Qualifications

The researchers are uniquely qualified to carry out this study because they work in the

King Library and are enrolled at SJSU for their Masters degree in Information and Library

Science. The researchers are concerned about the ability of students (specifically

undergraduates) to use the library’s OPAC to it’s fullest potential and therefore benefit the

students themselves to the fullest potential. As employees of the library, they have first-hand

knowledge of and interest in the potential uses of this study.

Significance of This Study

10


This study will benefit not only the King Library and those making decisions regarding

the library’s OPAC, but will also have broader benefits to librarians in aiding students in their

search endeavors and designers of next-generation OPACs in general. Librarians will be able to

better understand how undergraduate students are using the OPAC as well as where they are

having difficulty in understanding how to use the OPAC and will then be able to better serve

their needs as users. It will also benefit librarians who are creating library orientation tutorials

for undergraduate students. OPAC designers will benefit in a similar way: they will be able to

see where their systems are failing or meeting the specific needs of the end users.

Summary

Online public access catalogs are a student’s gateway to the wealth of information and

resources housed by academic libraries. As the information seeking needs of students change, so

must the vehicles that provide that information. Increasingly, traditional OPACs are no longer

meeting the information needs of their users. They are considered antiquated and are not

intuitive or readily learnable for a novice user. As Safley and Montgomery (2011) stated: “Many

libraries have grappled with making their online catalogs more accessible and relevant to their

users” (p. 164). A growing trend is focused around developing and implementing next-

generation OPACs to solve these issues. Next-generation OPACs look and function more like a

typical web based search engine, something younger users are more familiar with. It is

important that the same issues that users have with traditional OPACs are addressed and

remedied with next-generation OPACs in order for users to be capable of using them to their full

potential.

This study proposal seeks to discover what those issues are. By finding what,

specifically, makes both traditional and next-generation OPACs easy or difficult to use from an

11


undergraduate student’s perspective, this study can become an instruction manual for OPAC

designers and librarians who work with OPACs. This study will illustrate not only how

undergraduate students are searching via OPACs (through back end tracking), but also how they

feel about the experience (through their survey responses). The combination of those two

elements is essential to developing OPACs that serve user needs.

There have been plenty of studies on how traditional OPACs are used and what issues

users have with them. Next-generation OPACs are still new enough that the research is in its

beginning stages. It’s important to build upon the research that has already been conducted with

traditional OPACs and continue to study similar usability factors with next-generation OPACs.

Although this study does not claim to be exhaustive, it will most assuredly add to the research

literature already present. Besides adding to the current state of OPAC research, this study aims

to fill in some of the missing information regarding specifically how both types of OPACs are

failing to meet user needs. The intention the actual search habits of the participants with their

thought and feelings regarding their searches. By doing so, a more complete picture of how and

why users interact with OPACs the way they do and what is either working or not working

during that interaction will be illustrated.

This study will provide information on how to keep OPACs relevant in a search

environment with many other options. If users were able to as easily search an OPAC for

information as they do an Internet search engine, then OPACs will remain a viable option for

them to use. With the emergence of next-generation OPACs we are at a point where this is

possible and the benefits for scholarly research and quality information seeking behavior patterns

are great. As Knievel, Wakimoto and Holladay (2009) stated: “If libraries want their catalogs to

12


remain relevant to users, it is important that catalog designs change over time to reflect the way

that users are comfortable using the systems” (p. 457).

13


Reference List

Dinet, J., Favart, M., Passerault, J-M. (2004). Searching for information in an online public

access catalogue (OPAC): the impacts of information search expertise on the use of

Boolean operators. Journal of Computer Assisted Learning 20(5), 338-346.

doi: 10.1111/j.1365-2729.2004.00093.x

Emanuel, J. (2011). Usability of the Vufind next-generation online catalog. Information

Technology and Libraries 30(1), 44-52. Retrieved from:

http://web.a.ebscohost.com.libaccess.sjlibrary.org/ehost/pdfviewer/pdfviewer?

vid=7&sid=e0f9d0d2-d2be-49fe-916d-e279534fa6c1%40sessionmgr4001&hid=4101

Hessel, H. & Fransen, J. (2012). Resource discovery: comparative survey results on two catalog

interfaces. Information Technology and Libraries 31(2), 21-44. Retrieved from:

http://web.a.ebscohost.com.libaccess.sjlibrary.org/ehost/pdfviewer/pdfviewer?

sid=e0f9d0d2-d2be-49fe-916d-e279534fa6c1%40sessionmgr4001&vid=13&hid=4101

Johnson, F. & Craven, J. (2010). Beyond usability: the study of functionality of the 2.0 online

catalogue (OPAC). New Review of Academic Librarianship, 16(2), 228-250. doi:

10.1080/13614533.2010.511845

Knievel, J., Wakimoto, J.C. & Holladay, S. (2009). Does interface design influence catalog use?

A case study. College & Research Libraries, 70(5), 446-459. Retrieved from:

http://crl.acrl.org/content/70/5/446.full.pdf+html

Liu, Z. & Luo, L. (2011). A comparative study of digital library use: factors, perceived

influences and satisfaction. The Journal of Academic Librarianship 37(3), 230-236.

Retrieved from: http://www.sciencedirect.com.libaccess.sjlibrary.org/science/article/pii/

S0099133311000292

14

http://www.sciencedirect.com.libaccess.sjlibrary.org/science/article/pii/S0099133311000292

http://www.sciencedirect.com.libaccess.sjlibrary.org/science/article/pii/S0099133311000292

http://web.a.ebscohost.com.libaccess.sjlibrary.org/ehost/pdfviewer/pdfviewer?vid=7&sid=e0f9d0d2-d2be-49fe-916d-e279534fa6c1@sessionmgr4001&hid=4101

http://web.a.ebscohost.com.libaccess.sjlibrary.org/ehost/pdfviewer/pdfviewer?vid=7&sid=e0f9d0d2-d2be-49fe-916d-e279534fa6c1@sessionmgr4001&hid=4101


Mi, J. & Weng, C. (2008). Revitalizing the library OPAC: interface, searching, and display

challenges. Information Technology and Libraries 27(1), 5-22. Retrieved from:

http://www.ala.org/lita/ital/sites/ala.org.lita.ital/files/content/27/1/mi.pdf

Safley, E. & Montgomery, D. (2011). Oasis or quicksand: implementing a catalog discovery

layer to maximize access to electronic resources. The Serials Librarian, 60, 164-168. doi:

10.1080/0361526X.2011.556028

Yang, S. Q. & Hofmann, M.A. (2010). Next generation or current generation? A study of the

OPACs of 260 academic libraries in the USA and Canada. Library Hi Tech, 29(2), 266-

300. doi 10.1108/07378831111138170

Appendix A

Dear Student,

15

http://www.ala.org/lita/ital/sites/ala.org.lita.ital/files/content/27/1/mi.pdf


Hello! You have been randomly selected to participate in a usability study. This study is

looking to explore the use of the King Library’s new and old catalogs in a comparison study.

Your participation would be completely voluntary as well as anonymous. Your name will not

appear in any of the study findings and volunteering for the study will not result in any academic

reward or penalization.

If you are interested in partaking in this study, please send us a response to the email

address provided below no later than January 15, 2014. We would like to thank you in advance

for taking the time to consider participation.

Sincerely,

Darcy Cummings

[email protected]

Appendix B

16

mailto:[email protected]


Complete the following search exercises using the King Library’s old catalog and then repeat the

tasks using the King Library’s new catalog.

1. Find the book “Guns, Germs, and Steel” by Jared Diamond.

2. Find the spoken CD “The World According to Garp” by John Irving.

3. Find the book “Bridge to Terabithia” by Katherine Paterson that was published in 1997.

4. Find all books authored by Kurt Vonnegut available specifically in the King library.

5. Find a book related to drugs and poverty published in the last 2 years.

6. Find the DVD of “The Tempest” directed by Julie Taymor.

7. Find 4 articles or books for which “Alice in Wonderland” is the subject.

8. Find 4 books for which Evelyn Waugh is the subject.

9. Find the book “The Wind in the Willows” located in the Children’s Collection of the

library catalog.

10. Find all the movies based on books written by Kazuo Ishiguro.

Appendix C

17


Classic (traditional) OPAC search page:

18


New (next-generation) OPAC search page:

Appendix D

19


Please rate your experience using the old and new King Library catalogs. For questions with a 1-

10 scale: one is the lowest and 10 is the highest level of satisfaction or ease of use. Please circle

the number that you feel corresponds with your experience during the search exercises. For

questions that do not already provide an answer, please let us know your personal opinion(s).

1. How easily were you able to find each item or items in the search exercises?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

b. New catalog: 1 2 3 4 5 6 7 8 9 10

2. How easy was it for you to navigate from page to page and also back page?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

b. New catalog: 1 2 3 4 5 6 7 8 9 10

3. Was the overall design of the catalogs (colors, fonts, placement of links, etc.) appealing

to you?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

b. New catalog: 1 2 3 4 5 6 7 8 9 10

4. Search limiters provide parameters for your search by specifying date ranges, formats,

and type of search such as author, subject or keyword. Did you find the search limiters

(i.e. date ranges, author, subject, format, etc.) to be effective in narrowing your search?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

b. New catalog: 1 2 3 4 5 6 7 8 9 10

c. I did not use search limiters

5. Was it is easy to find the necessary limiters for your searches?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

20


b. New catalog: 1 2 3 4 5 6 7 8 9 10

c. I did not use search limiters

6. Did you find the ability to limit your search before the results page or after the results

page easier?

a. Old catalog: 1 2 3 4 5 6 7 8 9 10

b. New catalog: 1 2 3 4 5 6 7 8 9 10

c. I did not use limiters

7. If you did not use limiters in your searches, would you briefly explain why?

a. _____________________________________________________

8. Did you find any of the search features frustrating or not easily understood (please

specify)?

a. Old catalog: _________________________________________

b. New catalog: ________________________________________

9. Did you find any of the search features particularly helpful in your searches (please

specify)?

a. Old catalog: _________________________________________

b. New catalog: ________________________________________

10. Given the choice, which catalog would you prefer to use?

a. Old catalog

b. New catalog

c. I would not prefer either catalog

11. Do you have any other comments about the ease of use or lack thereof for either catalog?

a. Old catalog: _______________________________________________________

21


b. New catalog: ______________________________________________________

22

Documents

Cummings Research Proposal