View
10
Download
0
Category
Preview:
Citation preview
Kristen A. Bullard. Virtual reference service evaluation: An application of unobtrusive research methods and the Virtual Reference Desk’s Facets of Quality for Digital Reference Service. A Master’s Paper for the M.S. in L.S. degree. April, 2003. 80 pages. Advisor: Claudia Gollop.
This paper focuses on the quality of virtual reference service in public institutions
classified by the Carnegie Classification as Doctoral/Research – Intensive and
Doctoral/Research – Extensive universities. Virtual reference service for ten different
universities was evaluated by utilizing the Virtual Reference Desk organization’s Facets
of Quality for Digital Reference. Using unobtrusive methods the investigator posed the
same reference question to seven of the ten sites; investigating the following Facets of
Quality: “prompt turnaround, instructive, and interactive”. All ten universities’ Web sites
were evaluated by the following Facets of Quality: accessible, publicize, clear response
policy, privacy, and access to related information. This study finds not only that the
same issues of reference quality that exist in other methods of reference delivery also
exist in virtual reference; but that there is a need for further standardization in virtual
reference and a need for specific adaptations to the Facets of Quality for Digital
Reference to reflect the specialized nature of virtual reference and the evaluation of
virtual reference.
Headings:
College and university libraries – Reference services - Evaluation
Reference services – Evaluation
Reference services – Automation – Evaluation
Reference services – Standards
Carnegie Foundation for the Advancement of Teaching
Virtual Reference Desk
VIRTUAL REFERENCE SERVICE EVALUATION: AN APPLICATION OF UNOBTRUSIVE RESEARCH METHODS AND
THE VIRTUAL REFERENCE DESK’S FACETS OF QUALITY FOR DIGITAL REFERENCE SERVICE
by Kristen A. Bullard
A Master's paper submitted to the faculty of the School of Information and Library Science of the University of North Carolina at Chapel Hill
in partial fulfillment of the requirements for the degree of Master of Science in
Library Science
Chapel Hill, North Carolina
April 2003
Approved by:
___________________________
Advisor
1
Table of Contents
Table of Contents................................................................................................................ 1
Introduction......................................................................................................................... 3 Purpose of the study........................................................................................................ 4
Significance of the Study ................................................................................................ 5
Organization of this paper............................................................................................... 5
Definitions........................................................................................................................... 6 Digital Reference ............................................................................................................ 6
Carnegie Classification ................................................................................................... 7
Virtual Reference Desk’s Facets of Quality for Digital Reference ................................ 8
Literature Review.............................................................................................................. 10 Virtual reference service overview ............................................................................... 10
Unobtrusive methods in reference evaluation............................................................... 14
Reference service evaluation ........................................................................................ 16
Asynchronous Reference Evaluation........................................................................ 22 Virtual Reference Evaluation.................................................................................... 24
Methodology..................................................................................................................... 26 Introduction................................................................................................................... 26
Study Parameters .......................................................................................................... 27
Reference Question....................................................................................................... 28
Data Collection ............................................................................................................. 29
Test Procedures............................................................................................................. 32
Data Analysis ................................................................................................................ 33
Results............................................................................................................................... 34 Expected Answers......................................................................................................... 34
Facet Scoring ................................................................................................................ 35
Individual Site Results .................................................................................................. 37
2
Site 1 Results............................................................................................................. 38 Site 2 Results............................................................................................................. 39 Site 3 Results............................................................................................................. 41 Site 4 Results............................................................................................................. 42 Site 5 Results............................................................................................................. 44 Site 6 Results............................................................................................................. 45 Site 7 Results............................................................................................................. 47 Site 8 Results............................................................................................................. 48 Site 9 Results............................................................................................................. 49 Site 10 Results........................................................................................................... 51
Analysis............................................................................................................................. 52 Hours of operation ........................................................................................................ 53
Facet Results ................................................................................................................. 57
Facet 1 – Accessible.................................................................................................. 58 Facet 2 – Prompt Turnaround ................................................................................... 59 Facet 3 – Clear Response Policy............................................................................... 60 Facet 4 – Interactive.................................................................................................. 61 Facet 5 – Instructive.................................................................................................. 61 Facet 8 – Private........................................................................................................ 62 Facet 10 – Provides Access to Related Information ................................................. 62 Facet 11 – Publicize .................................................................................................. 62
Discussion..................................................................................................................... 63
Future Investigation ...................................................................................................... 66
Conclusion ........................................................................................................................ 67
Bibliography ..................................................................................................................... 68
Appendix........................................................................................................................... 74 Site Demographics Form .............................................................................................. 75
Site Evaluation Form .................................................................................................... 76
Virtual Reference Interaction Form.............................................................................. 77
Virtual Reference Evaluation Form.............................................................................. 78
3
Introduction The use of the reference desk, phone, electronic mail, Web forms, chat, Web
tutorials and videos, subject guides, and even videoconferencing include the many ways
people can satisfy their information needs in academic libraries. With so many ways for
people to seek reference help what makes them choose one format over the other? Do all
the reference service formats provide the same quality of service? One librarian
interviewed about virtual reference (VR) indicated that “people tend to avoid email for
their general questions because they do not think they will get a timely answer.”
(Sessoms, 2002) While time may be one factor in personal preference other factors
include convenience, level of comfort and usability.
Users prefer different methods for different reasons. While some users dislike the
anonymity of digital reference, others prefer the added privacy digital reference offers.
This study looks at the VR services that are available in research universities and
evaluates the need for standardization of VR. Using unobtrusive study methods and the
Virtual Reference Desk’s Facets of Quality for Digital Reference Services this study
finds not only that the same issues of reference quality that exist in other methods of
reference delivery also exist in virtual reference; but that there is a need for further
standardization in virtual reference and a need for specific adaptations to the Facets of
Quality for Digital Reference to reflect the specialized nature of virtual reference and the
evaluation of virtual reference.
4
Purpose of the study
The purpose of this study is to explore the quality of VR services available in
academic institutions classified by the Carnegie Classification as public
Doctoral/Research – Intensive and Doctoral/Research – Extensive. Particular emphasis is
placed on how the reference interview is used in the digital environment, as well as the
quality and accuracy of the answers given. The study also determines the level of
standardization of similar digital reference studies and found the Virtual Reference Desk
(VRD) Facets of Quality for Digital Reference Services to be the most comprehensive
tool for evaluation of digital reference services. However, this tool is designed to
primarily evaluate asynchronous digital reference services and is not thoroughly adapted
to the evaluation of VR services. By adapting and utilizing this tool the investigator
hopes to start identifying which parts of the VRD Facets of Quality should be revised for
the evaluation of VR.
The following research questions are explored in this study:
• Is the Virtual Reference Desk’s Facets of Quality for Digital Reference an effective
means for evaluating virtual reference service?
• What is the current level of effectiveness of virtual reference service including hours
of operation; accuracy of responses; quality of reference interview; ease of access to
the service and ease of use; availability of supporting materials such as policy
statements, service documentation, complementary digital resources, and privacy
policies?
5
Significance of the Study
This research will contribute to the small but growing literature on VR, while
applying unobtrusive research methods, a format utilized for decades in reference service
evaluation. This study introduces some of the issues involved in using unobtrusive
methods to evaluate VR. Additionally, this study discusses the importance of adapting
the Virtual Reference Desk’s Facets of Quality for Digital Reference to further improve
the consistency and quality of virtual reference service.
Organization of this paper
The Introduction contains an overview of the paper including the purpose and
significance of the study. The frequently used terms in this field are explained in the
Definitions portion; including terms relating to digital reference, the Carnegie
Classification, and the Virtual Reference Desk’s Facets of Quality for Digital Reference.
The Literature Review provides a summary of background information and current
research trends in reference service evaluation, digital reference, and unobtrusive study
methods. The Methodology discusses the steps taken in the study such as site selection
and VR interaction; and the procedures taken to assign values to the data. The Results
report the data collected for all Web sites investigated. The Analysis synthesizes the data
and discusses the findings and limitations of the study as related to the Facets of Quality
and presents recommendations for further investigation. The Conclusion provides a
summary of the study, its findings, and what may be determined by those results.
6
Definitions
Digital Reference
There is great ambiguity in the terms used to refer to digital reference and all of
its components. However, in an effort to establish some consistency within this paper the
following terms are used and are defined as noted:
• Digital reference – The term digital reference is “used to describe the entire world of
electronic reference.”(Riggs, 2002, p. 7) This includes tutorials, subject guides, and
asynchronous and synchronous formats.
• Asynchronous digital reference – The term asynchronous digital reference refers to
formats of digital reference service that do not occur in real-time. These include
electronic mail and Web forms. Electronic mail uses one or more email addresses to
provide the user a point of contacting reference help. Web forms are forms placed on
the library web site that prompt the user for details about their information need. The
replies and further reference interview communication is sent via email for both email
and Web forms.
• Synchronous digital reference – The term synchronous digital reference refers to
formats of digital reference that occur in real-time. The literature refers to this
service as virtual reference, chat reference, instant messaging reference, live-virtual
reference, e-reference, live reference, and even digital reference. Throughout this
paper synchronous digital reference will be referred to as virtual reference (VR).
Tunender (2002, p. 5-6 ) divided VR by the three types of software available:
7
1. “Chat. Chat software, such as AOL’s Instant Messenger, allows the simple exchange of text messages.”
2. “Remote Control Software (RCS)(e.g. Convey’s OnDemand, Expertcity,Inc.’s Desktopstreaming). Remote control software allows the librarian to control the patron’s browser as well as communicate using chat.”
3. “Web Contact Center Software (e.g. LSSI’s Virtual Reference Software and MCLS’s 24/7 Reference). Web contact center software allows the patron to ‘call’ a librarian simply by clicking on a hyperlink on the library’s Web site. The two can then chat and co-browse the Web (share the same Web pages.)”
Carnegie Classification
Established in 1973, “the 2000 Carnegie Classification includes all colleges and
universities in the United States that are degree-granting and accredited by an agency
recognized by the U.S. Secretary of Education. The 2000 edition classifies institutions
based on their degree-granting activities from 1995-96 through 1997-98.” (Carnegie,
2000) This study focuses on the top two classifications: Doctoral/Research Universities-
Extensive and Doctoral/Research Universities-Intensive. They are defined by Carnegie
as follows:
Doctoral/Research Universities-Extensive: These institutions typically offer a wide range of baccalaureate programs, and they are committed to graduate education through the doctorate. During the study period, they awarded 50 or more doctoral degrees per year across at least 15 disciplines.
Doctoral/Research Universities-Intensive: These institutions typically offer a wide range of baccalaureate programs, and they are committed to graduate education through the doctorate. During the period studied, they awarded at least ten doctoral degrees per year across three or more disciplines, or at least 20 doctoral degrees per year overall.
Out of the total 3,942 classified institutions 151 are Extensive (3.8%) and 110 are
Intensive (2.8%). Doctoral/Research Universities constitute seven percent of all
classified institutions. (Carnegie, 2000)
8
Virtual Reference Desk’s Facets of Quality for Digital Reference
Maxwell’s (2002) report on “Establishing and Maintaining live online reference
service” explains the Virtual Reference Desk document entitled “Facets of Quality for
Digital Reference”, Version 4, October 2000. The Facets of Quality, originally created in
1999, is intended to be a guide for the creation and maintenance of digital reference
service. The Facets of Quality “includes all major online reference assessment areas,
with three levels of attainment in each: base, current practice, and goal level….’Facets’ is
divided into two categories: user transaction and service development and management.”
(Maxwell, 2002, p. 33-34) These are the VRD Facets of Quality and their definitions as
summarized by Maxwell:
Facets for user transaction:
1. Accessible: Digital reference services should be easily reachable and navigable by an Internet user regardless of equipment sophistication, physical disability, or language barrier.
2. Prompt turnaround: Questions should be addressed as quickly as possible. Actual turnaround time depends on a service’s question-answer policy and available resources (such as staffing and funds).
3. Clear response policy: Clear communication should occur either before or at the start of every digital reference transaction to reduce opportunities for user confusion and inappropriate inquiries.
4. Interactive: Digital reference services should provide opportunities for an effective reference interview, so users can communicate necessary information to experts and the experts can clarify vague user questions.
5. Instructive: Digital reference services provide access to current information and expertise. Quality digital reference services offer more to users than straight, factual answers; the services guide users in subject knowledge as well as information literacy.
Facets for service development and management:
9
6. Authoritative: Experts of a digital reference service should have the necessary knowledge and educational background in the service’s given subject area or skill to qualify as an expert. Specific levels of knowledge, skill, and experience are determined by each service and its related discipline or field.
7. Trained Experts: Services should offer effective orientation or training processes to prepare experts to respond to inquiries using clear and effective language and following service response policies and procedures. Training of information specialists is one of the most important aspects of planning and operating a digital reference service.
8. Private: All communications between users and experts should be held in complete privacy.
9. Reviewed: Digital reference services should regularly evaluate their processes and services. Ongoing review and assessment help ensure quality, efficiency, and reliability of transactions as well as overall user satisfaction.
10. Provide access to related information: Besides offering direct response to user questions, digital reference services should offer access to supporting resources and information. Services can reuse results from question-answer exchanges in resources such as archives and frequently-asked questions (FAQs).
11. Publicize: Services should inform potential users of the value that can be gained from use of the service. A well-defined public relations plan can ensure that services are well-publicized and promoted on a regular basis. Publicity should not create more demand than the service has capacity to handle. (Maxwell, 2002, p. 34-35)
The Virtual Reference Desk organization’s discussion about the need for
standardization in digital reference service states the following:
To be included in the VRD Network all services must fulfill the base level of requirements for each facet of quality. The current practice level represents a wide range of possibilities and reflects the current state of services in the network. The goal provides an optimal level to which all AskA services in the network will strive to achieve over time. (VRD, 2000)
It is clear that there is a need for standards as digital reference evolves from a handful of AskA services and a set of experiments in libraries to a common means of interacting with users. These facets of quality provide an example of standards of operation for digital reference services in a cross-domain environment. It is hoped that these standards will be adopted in growing reference consortia, or used as a model for the development of new digital reference standards. (VRD, 2000)
10
Literature Review
Few sources exactly fit the focus of this study, which is evaluating virtual
reference service in academic libraries using unobtrusive methods; therefore, this review
of the literature includes a discussion of general trends in virtual reference; unobtrusive
methods for reference evaluation; and reference service evaluation including
asynchronous reference and virtual reference.
Virtual reference service overview
Because this is such a new field of study there is a need for standardization not
only in the terms used (as discussed in the introduction), but also in the software, goals,
service, and evaluation. Maxwell (2002) states that “once these standards are proposed
and adopted, digital reference likely will take the next major step in its development.”(p.
36) Exemplary of the need for standardization is a discussion summarized in the 2000
conference proceedings for the VRD conference, some of the main points and key issues
of the conference include: “reference service standards…service standards and quality
are proving very difficult in digital reference,” “software…standards need to be further
developed for questions and answers if data is to be shared or automation used to its full
potential,” “who pays…for organizations who have a mandate to provide public
information,” and “partnerships…to identify subject strengths.” (Missingham, 2000, p.
209).
Even though there has been some concern for librarians and reference service
needing to undergo major changes to accommodate VR, Qayyum (2002) states “while
11
search tools have changed in the recent years, user queries that need to be addressed by
librarians have undergone very little change.” (p. 21). The methods for delivering the
services, however are changing, and offering a greater flexibility for user preferences in
this new communication method.
Many papers written on VR include papers discussing the status of VR service;
frequency of usage, types of questions, hours, etc…. Tennant (2003) is one paper written
to discuss the status of virtual reference and poses the question of whether low usage
statistics are a result of low awareness or low interest in the service. Tennant states that
“although this question is of primary importance…, we need more experiences and better
measures before we can determine where the problems lie.” (p. 38)
Janes (2002) presents an in depth study which evaluates academic and public
librarians’ attitudes and experiences with technology including email and VR services.
In a table summarizing the “use of technology by size and type of library” for academic
libraries, Janes’ research reveals the following digital reference service usage:
Asynchronous reference services - 89% offer E-mail, 60% offer Web forms, VR – 4 %
offer instant massaging, 6% offer chat, 5% offer videoconferencing, 2.6% offer
MOO/MUD. (p. 554)
Peters (2002) discusses the current and best practices for the role of consortia in
electronic reference services. There is no study being conducted or reported, instead this
paper is a tool for researchers and service providers. The broad topic areas for the role of
consortia include negotiation of license agreements, collaborative e-reference servers,
12
providing software for member libraries, selection processes of electronic reference,
communication and/or education of software providers, and referral services.
Tenopir (2002) conducted a diffusion of innovation study; this study consisted of
an analysis of four surveys sent to ARL academic libraries in the past ten years. The
findings include the modes of reference offered, the types of resources being utilized, and
the rate of adoption of new methods of reference. The findings state that:
In addition to regular drop-in reference desk services and telephone reference, all but one of our respondents now offer e-mail reference and all but three do reference service by appointment. The newest option is real-time online reference-mostly through synchronous text chat-now offered by more than 28 percent (twenty libraries) of our respondents….Still, coming in to the reference desk is the most heavily used form of reference services in all of the libraries that responded to that question. (p. 271)
Fifty-eight of the seventy libraries made estimations about the types of resources they
utilize. “These reference librarians use a wide variety of resources to answer reference
questions but speculate that the library’s own online catalog and fee-based (commercial)
online databases result in the greatest number of reference answers.” (Tenopir, 2002, p.
272)
Yet another type of paper frequently written about VR includes papers from
universities sharing the successes, challenges and procedures involved with implementing
VR. These papers are presented in a variety of methods including conversational
narrative and scientific study. Broughton (2001) is an informal discussion of Bowling
Green State University’s process of selecting VR software and the experiences of
implementation. This paper includes an interesting comparison of a teenager using
13
instant messaging to communicate with friends (their expectations and habits) and
librarian anxieties with the users’ expectations of VR. Broughton states:
[The teenager] was multitasking…listening to music, chatting in two or three different conversations, and surfing the Web. The time lag didn’t bother her at all. She was doing other things in between thoughts in one particular conversation, and if she had another thought before she received her friend’s reply, she just went ahead and sent it. [This] certainly is not natural to [librarians], but many of the students don’t seem to even notice [a lapse in conversation]. (p. 28)
Boughton goes on to explain that
Despite the strangeness of communicating via chat, it didn’t take long to get accustomed to it. We discovered that users tended to send many shorter messages rather than one long paragraph. The transcripts don’t make complete linear sense, but while you’re in the conversation, it’s understandable. It certainly eases the anxiety of empty waiting time. The users didn’t seem to be as bothered by the length of time it took (to send, have the other person read, and then reply and send back) as we were. Eventually, we got used to doing something else too (like directing someone to the pencil sharpener, looking up items in the catalog, etc.). (p. 30)
One concern that this statement raises is that expecting librarians to multi-task too much
will compromise the quality of service and eliminate many of the “real-time” and
“instant” characteristics of this form of reference service. While a further study of teen
chat techniques could potentially help improve librarian VR communication skills, it is
not a good idea to start encouraging librarians to conduct too many reference interactions
at one time.
Foley’s study is an excellent example of the wide range of papers about how
different libraries have explored VR. This paper discusses the process of selecting
software, implementation, staffing and marketing VR service for a pilot project at the
General Libraries of the University of Buffalo. It goes on to evaluate “user
14
demographics, satisfaction levels, usage statistics, patron comments, and librarian
feedback.” (Foley, 2002, p. 36)
Unobtrusive methods in reference evaluation
Many studies of reference quality have focused on reference service outcomes….One series of measures relies upon unobtrusive testing and has been used most extensively to evaluate the accuracy of answers. Other studies rely primarily upon user judgments of success or upon librarian judgments of success. Another set of measures relies upon observational data collected by librarians but analyzed by experts not directly involved in the reference transactions. Still other measures rely upon user judgments of outcomes, but also collect librarian judgments of outcomes from the same reference transaction. (Whitlatch, 2001, p. 7)
Unobtrusive methodology has been used to evaluate the accuracy and quality of
reference service for many years. Studies and articles about using unobtrusive methods
in reference evaluation are included in this section; the next section, Reference Service
Evaluation, will include studies and articles that utilize this methodology. “The
unobtrusive method…was applied to reference service for the first time by Terence
Crowley in 1967….The basic theme of the unobtrusive study of reference has always
been to (1) ask a library staff member a query, posing as a real client, and (2) judge the
response.” (Childers, 1990, p. 27). “…Some have claimed that the findings of
unobtrusive reference studies indicate that the quality of reference work, generally, is
little better that at the 50 percent level; others have claimed that the studies were so
limited in scope that such broad claims about reference work in general were
misleading.”(Childers, 1990, p. 33) In an article about utilizing the unobtrusive method
in library reference evaluation Crowley summarizes the major works using unobtrusive
methods since his first study in 1967. The article discusses the “guidelines for better
15
unobtrusive studies; and what effect…there has been on the profession.” (Crowley, 1985,
p. 59) This article also contains an excellent table containing a data comparison of 14
different unobtrusive studies.
Two people who have had an impact on the use of unobtrusive methods in library
reference evaluation are Herndon and McClure. In a 1986 study Herndon and McClure
report that “participants in the study answered 62 percent of the questions correctly and
38 percent of the questions incorrectly…. The most frequent reason for an incorrect
answer was that library staff gave wrong data (…64 percent).” 20.1 percent of the “staff
responded with ‘don’t know’ and terminated the search without suggesting any referral.”
15.4 percent “claimed that the library did not own a source that would answer the
question, when, in fact, it did own such source.” (Herndon, McClure, 1986, p. 38-39) In
their 1987 article Herndon and McClure summarize the authors’ past work, stressing both
the importance of unobtrusive testing for evaluating reference services and the
importance of improving the quality of reference services. The authors submit a list of
reference staff shortcomings including the “55% rule” of reference accuracy and
librarians stating “I don’t know” as an answer to reference queries. (p. 69-70)
Dilevko (2000b) conducted a follow-up to the study done by Herndon and
McClure in the early 1980s evaluating government documents reference service using
unobtrusive methods. Some of her points include:
o By whose standards are service quality and professional competence to be judged? Should the public have something to say about it, or should it be an internal professional matter? (p. 6-7)
16
o Unobtrusive reference testing is a valid measure of both knowledge and accuracy, even if it is confined to the type of unambiguous fact-based questions library staff say is a very small part of their total work load at reference desks. (p. 11)
o Accordingly, one of the main benefits of unobtrusive testing is that it allows librarians, individually and collectively, to debate what it means to be a professional, what is means to provide good service. (p. 18)
Whitlatch’s 1989 article discusses the “validity and assumptions regarding
unobtrusive studies.” She concludes by stating that unobtrusive test questions should be
designed to represent “all types of queries” and measuring the correctness of answers
should be supplemented “with other measures of reference performance.” (Whitlatch,
1989, p. 181) This study attempts to utilize the second recommendation by utilizing the
Facets of Quality for Digital Reference to evaluate more than just correct answers in the
evaluation of each site’s virtual reference.
Reference service evaluation
Many of the studies which evaluate reference services utilize unobtrusive methods
as well as a number of other methods. The articles discussed in this section include the
evaluation of reference service including asynchronous digital reference and virtual
reference. These articles include implications for all aspects of the reference encounter.
One article discusses the ten strategies of negative closure employed by librarians
during the reference encounter. (Ross & Dewdney, 1998) The reasons for these tactics,
Ross & Dewdney feel, is the increase in reference service use, a very busy reference desk
causes librarians to use such strategies as negative closure to increase speed and serve
more patrons. (p. 152-153)
17
Dilevko (2000a) discusses similar trends in the reference encounter that resulted
from an unobtrusive study on government documents reference. Dilevko’s study found
librarians utilizing “helpful behaviors” including “real interest about user needs”, “use of
multiple sources”, and “collaboration with other staff.” However the article focuses on
the negative reference encounters stating that “on the whole, [users] were disappointed in
how they were treated…. For example, reference personnel were criticized for providing
numerous unmonitored referrals, telling users that the questions were too difficult, and
for not being sufficiently knowledgeable about government documents.” (Dilveko,
2000a, p. 299) Additionally, this study by Dilevko (2000a) finds that librarians are
utilizing the same negative closure tactics when utilizing Web resources. The segment of
the paper entitled “The unmonitored referral in the electronic age” includes findings such
as:
o Staff assume that the mere act of providing a single gateway Web address constitutes good service. However, electronic sources of information are no less complex than print sources, and not all patrons…may be adept at navigating Web pages, let alone understanding the structure and authority of Web documentation. (p. 305)
o … it would appear that a Web site address is being used as a shortcut response intended to ’get rid of’ a patron, with no follow-up and no suggested alternatives should the original Web address fail to meet an information need. (p. 306)
o The Web seems to be functioning as a ‘one-size-fits-all’ source. (p. 307)
o In sum, the unmonitored referral occurs both with traditional print sources and electronic sources.”…”The web is imbued with a mystique of omnipotence, and the mere mention of it may induce, in some patrons, a feeling that immediate help is moments away. But, as one proxy states, this tactic may be a sign of a mere fundamental problem, namely, reference personnel who are attempting to hide their lack of skills. (p. 306-307)
18
Due to the number of studies focusing on the negative aspects of reference service
Whitlatch (2001) feels that “assessing reference services requires adopting a set of
performance standards. A performance standard is a description of the level of
achievement expected of a person or organization.” (p. 1) In the section entitled “Quality
of the Reference Service Process” Whitlatch states reference service users can not
effectively evaluate the content of the answers received in the reference interview;
however they can evaluate the “process of receiving service.” (p. 3-4) Topics in this part
include: “process of selecting answer sources, quality service measures as applied to
reference services; the impact of nonverbal communication; the user’s willingness to
return as a key measure of reference success; librarian behaviors; the interpersonal
dynamics between librarian and user; and interviewing skills.” (Whitlatch, 2001, p. 3-4)
In the section of Whitlatch’s (2001) article entitled “Quality of Reference Service
Products” Whitlatch states that “evaluating reference service products or outcomes
involves assessing the quality of the actual answer of information the user obtained as a
result of the service. This set of standards emphasizes the benefit or value of the service
to the user.” (Whitlatch, 2001, p. 6) Whitlatch also explains:
Among the performance standards often used to measure effectiveness of service are (1) economic – e.g., cost effectiveness, productivity measures; (2) service process – e.g., measures of satisfaction with the service provided; (3) resources – e.g., measures of quantity and quality of materials, staffing, equipment, and facilities supporting the service; (4) service outcomes or Products – e.g., measures related to the quality of answers or information delivered. (p. 2)
Whitlatch (2000) also created a resource entitled Evaluating reference services: a
practical guide. This tool is intended to be a resource for library researchers to use as a
guide for developing a study on evaluating electronic reference services. It explains the
19
establishment of goals and objectives, and selecting a study method which matches the
goals and objectives. The study methods detailed in the article include surveys and
questionnaires, observation, interviews, and case studies.
Norlin (2000) conducted a study that combines three research methods; these
include survey, unobtrusive observation, and focus group. This study provides a
comprehensive evaluation of reference service. Findings resulted in a list of eight easy to
implement changes or improvements the library can make to improve the overall service.
This type of study can be implemented to evaluate digital services. (p. 552-553)
Bowman (2002) paraphrases ASCLA’s “The Reference Interview: A Common-
Sense Review,” by summarizing that an effective reference interview includes the
following:
• “Greet – look welcoming and animated; appear interested in the person and what he or she needs.”
• “Probe – restate the request; ask open[-ended] questions to get more information.”
• “Verify – restate a specific question by paraphrasing; ask if some sources have been checked and what was found.”
• “Locate Information – check appropriateness of level, depth of coverage; ask about format preferences; offer help with using the tool.”
• “Close – check if the question has been or can be answered; expressly offer additional help as needed.”
• “Follow-up – ascertain if the question has been completely answered; check to see if anything else is needed.” (p. 8)
One study discusses the important role of the librarian as the library public
relations representative. Tyckoson (1992) discusses the librarian’s accountability not
only to the user, but also to other librarians, library administrators, and the library
20
profession and elaborates on the different attributes that are present in effective reference
work. (p. 151) These attributes include:
Attributes of Availability • Positive Indicators
o The librarian arrives for reference duty on time. o The librarian makes eye contact with potential patrons. o The librarian monitors the desk while assisting other patrons.
• Negative Indicators o The librarian erects barriers at the reference desk. o The librarian works on paperwork while at the desk. o The librarian talks to colleagues or friends while at the desk.
Attributes of Communication Skills
• Positive Indicators o The librarian actively listens to the patron. o The librarian asks the patron open-ended questions. o The librarian restates the question for the patron. o The librarian explains the rationale for selecting the strategies and
materials used. • Negative Indicators
o The librarian interrupts the patron. o The librarian speaks in a rude or condescending tone. o The librarian does not ask clarifying questions. o The librarian provides a response without discussing the methods used
in achieving that response.
Indicators of Search Strategy Skills • Positive Indicators
o The librarian is able to break a query into its component parts. o The librarian is able to select the facet(s) of the query that will result in
the shortest search time. o The librarian makes use of the basic Boolean operators OR, AND, and
NOT. • Negative Indicators
o The librarian begins the search in a source that is too detailed or too general.
o The librarian uses only a single strategy and does not try alternate search paths.
o The librarian does not make use of colleagues for suggestions.
Attributes of providing individual attention to patrons • Positive Indicators
21
o The librarian takes the patron to the sources. o The librarian explains how to use reference tools. o The librarian checks back with patrons after they have been working
for a period of time. • Negative Indicators
o The librarian makes vague reference to areas or sources [unmonitored referrals].
o The librarian interrupts one patron to receive a telephone call or to work with another. (p. 168-172)
In their 2000 study Ross and Nielson investigated the “impact of the Internet on
reference service.” (p. 148) They found that the Internet has not changed reference
service with continued “problems include[ing] failure to conduct a reference interview,
unmonitored referral and failure to ask follow-up questions.” (p. 147) They define an
unmonitored referral as: “the staff member refers the user to a source, either inside or
outside the library, but does not take any steps to check whether the user eventually gets a
helpful answer.” (p. 150)
Yet another study which discusses the quality of reference service is by Dewdney
and Ross (1994). They found that “willingness to return (59.7 percent) and overall
satisfaction” (p. 217) with the service is directly related to the quality of reference service
and that these are “significantly related to the librarian’s behavior and the quality of the
answer.” (Dewdney and Ross, 1994, p. 217) A later study by Ross and Dewdney (1998)
focuses on “negative closure” in the reference interaction usually due to feeling rushed to
move on to the next user. Some strategies for “getting rid of the user” that are discussed
at length include:
1. “The librarian provides an unmonitored referral;
2. The librarian immediately refers the user somewhere else, preferably away;
22
3. The librarian implies that the user should have done something else first before asking for reference help;
4. The librarian tries to get the user to accept more easily found information;
5. The librarian warns the user to expect defeat;
6. The librarian encourages the user to abort the transaction voluntarily;
7. The librarian states explicitly that the search has reached a dead end;
8. The librarian claims that the information is not in the library or else doesn’t exist at all; and
9. The librarian goes off to track down a document but then never returns.” (p. 154-156)
Boyer makes the additional point that the library Web pages must play an
increasingly important role in the reference process, stating “libraries must strive to
create Web environments in which answers to the most frequently asked questions are
easy for patrons to find without having to contact the reference desk.” (2001, p. 122)
Asynchronous Reference Evaluation
Bushallow-Wilbur (1996) completed a study of people using email reference
service. The discussion included: who uses the service, the types of questions asked,
when and where the questions were asked, and why this method is used over other
methods. The study shows that the order of reference service format places email first,
desk second, and phone third. With participants stating “It is much easier for me to send
the message at the moment I have a question than it is to make a note to find the time to
go into the library.”(p. 367) A contradictory reason is “that often times their information
need was not at all urgent or pressing” (p. 367) and they found email a sufficient format
for their question. “Accuracy appeared to be related to both a perception that a question
23
expressed in written form would present a precise description of the information need and
the perception that the person answering the e-mail query would give an accurate answer
in writing.”(p. 367-368)
Bowman (2002) discusses the role of the library Web site in the virtual reference
interview. He argues that the library Web site is an important component of the overall
reference experience, serving as the first line of user inquiry for directional and ready
reference information. Better content and design of a library Web site enables users to
easily exhaust accessible Web resources; if they still have questions digital reference is
quickly available to assist in their inquiry.
White (1999) presents guidelines for evaluation and creation of digital reference
service, mainly Ask-A Librarian services through educational websites. The sections of
this “Framework for Analysis of an Electronic Question/Answer Service” include:
“Mission, Objectives, Statement of Purpose; Parameters of Service: Questions; Parameters of Service: Clients; Administration; Staffing and Training; Hardware and Software; Ease of Use, Instructions to Client; Responsibility to Clients; Query Form; Acknowledgement; Question Negotiation; Question-Answering Process; Response Guidelines; Coping with Demand; Archiving; Quality Control; Evaluation; External Recognition.” (p. 3)
In a later study, White (2001) focuses on the diffusion of innovation of asynchronous
reference service in academic libraries. Out of the 140 academic libraries in the study
White found that 45 percent offered digital reference service, this data refers only to
asynchronous reference service. (p. 175)
Janes, Carter, and Memmott (1999 ) conducted a Web site study of 150 academic
libraries to evaluate the number of institutions offering asynchronous digital reference
24
(45 percent) and examine “direct links from library home pages, ways in which users
submit questions, FAQ documents, policies, technological barriers, and the role of
institutional control.” (p. 145) They conclude by stating that “it appears that using the
Internet as a medium for the reference process in academic libraries is still in its infancy.”
(p. 149)
Virtual Reference Evaluation
Taher (2002a) evaluates the effectiveness of both the asynchronous reference
interview and the VR interview. He applies Ranganathan’s Five Laws of Library Science
to “the provision of reference services via the Internet, adapting them to be used as an
evaluation tool” (p. 25):
1. Information is for use 2. Every interviewee his/her information 3. Every information its interviewee 4. Save the time of the interviewee 5. Information is a living organism” (p. 26)
Taher’s 2002a article concludes by stating the ultimate goal of digital reference
evaluation: “to save the time of the interviewee,” needs a “clearly defined scope” and “a
holistic study, analyzing not only whether the interactive reference interview is beneficial
to the interviewee, but also whether it saves other resources.” (p. 32) In a separate article,
Taher (2002b) discusses an analysis of the virtual reference literature and resources found
up until March 2002.
Helman discusses the limitations of VR for more complex information needs by
stating “referrals have become a core part of Ask Us!-Live [digital reference service]
primarily because of the general reference nature of this service, but also due in part to
25
the limitations of working in a digital environment (no print resources, some questions
too complex, etc.)” (p. 93) Technical challenges are also discussed at length, including:
“Co-browsing databases and websites; limiting the service to the MIT community;
browser/platform compatibility; speed/reliability over the Internet; time (of
implementation); human resources (staffing); buy-in (of library administration and staff);
training; referrals; [and] technical support.”(2001, p. 87-93)
Lankes and Shostack’s 2002 study on VR attempts to contradict Peters’ findings
about the necessity of real-time reference. Peters’ argument (as stated by Lankes and
Shostack) is that real-time reference is more effective than asynchronous reference
service due to the instant nature of the service and the ability to perform a reference
interview through VR. Peters states that “delays of more than a few minutes significantly
diminish both the usefulness and use of reference service.” (Lankes & Shoctack, 2002, p.
350) Lankes and Shostack’s counter-argument is that asynchronous reference is just as,
if not more, effective than real-time reference service, stating that many users are not
concerned with the wait time involved to receive an answer. The study analyzes
AskERIC feedback responses from 1998 and 2000 and a Virtual Reference Desk AskA
service usage survey. As stated in the authors’ limitations (p. 352) this study’s main
weakness in formulating its rebuttal of Peters’ article is that they are comparing apples to
oranges. The types of users that use asynchronous reference service have different
expectations than real-time reference service users.
Ruppel and Fagan (2002) conduct a thorough survey of VR including a
comparison of the advantages and disadvantages of both VR and desk reference service.
26
They summarize these findings by stating VR suits some reluctant patrons “by offering
anonymity and quick help without showing the busyness of the librarian.”(p. 194)
However, Ruppel and Fagan find that VR “will not replace the traditional desk in the near
future. Library users still want help with physical materials, and explaining the
intricacies of database searching are much easier face-to-face.”(p. 194)
Kibbee, Ward, and Ma (2002) conducted a study on a pilot VR project; the
findings include an analysis of who uses the service, when the service is used, location of
links to the service on the Library’s Web site, questions asked, duration of the VR
sessions, accuracy and the reference interview, and users’ opinions of the service. The
authors’ summarize the study by stating:
In general, quick questions about the library and ready reference/fact-based questions proved the easiest to answer. For hard to find answers, users could receive a response via our e-mail service. The hardest types of questions were queries requiring basic bibliographic instruction – getting started on a research paper, needing instruction in using the online catalog or searching article databases, etc. Since the software we had did not have co-browsing functionality, it was very difficult to help these students, and often the best response we had was to get them started and recommend they come in to the library for more in-depth assistance. (p. 35)
Methodology
Introduction
This study evaluates the accuracy and quality of virtual reference (VR) service
using unobtrusive measurements. As Whitlatch (2001) states, “Unobtrusive observation
methods can…be used effectively in an electronic world. Reference questions can be
prepared and answers determined for factual types of questions.” (p. 213) In using
27
unobtrusive methods for this study the investigator posed as an undergraduate student at
seven different university VR services asking the same question during each reference
interaction. In addition to the VR interaction a total of ten university’s websites were
evaluated. The VR evaluation is based on the Virtual Reference Desk’s (VRD) “Facets
of Quality for Digital Reference Services (Facets of Quality)” (2000). While this VRD
document is based primarily on asynchronous reference service, it contains elements that
are present in all forms of reference services and should be expanded to more specifically
address VR services.
Study Parameters
Careful site selection included a number of parameters which were chosen to
eliminate as many variables as possible within the study; the parameters require each site
to (1) have the same Carnegie Classification, (2) use the same VR software, (3) not be a
branch library, and (4) not be a member of a VR consortium/collaboration.
The first parameter (1) - The schools had to be public universities classified as
Doctoral/Research University – Extensive or Doctoral/Research University – Intensive as
established by consulting the 2000 Carnegie Classification. Having the schools all
classified as public universities reduces the probability of restricted access to the VR
service.
The second parameter (2) - The sites selected all had to use the same VR
reference web contact center software. This parameter was established by consulting
several sources. The software company was contacted for a list of current academic
clients, and several websites that index library VR services (LiveRef, VRD, Teaching
28
Librarian, and Bernie Sloan’s Digital Reference Pages) were consulted as a cross
reference. Finally each university web site was visited to confirm VR service operability
and the type of software usage. Choosing only one type of software reduces the variables
associated with software functionality such as co-browsing.
The third and fourth parameters were established using the same resources as the
second. The third parameter (3) - None of the libraries could be a branch library, was
established to maintain a consistency of collection and user types. Finally, the fourth
parameter (4) - No VR reference collaborations. While there are numerous benefits to
library consortia (Peters, 2002), this parameter was established because most
collaborations of VR reference included libraries from more that one Carnegie
Classification thus not fitting the first site selection parameter. Out of the 266 public
universities in the Doctoral/Research-Extensive and Doctoral/Research-Intensive
Carnegie Classifications 10 met the parameters of this study.
Reference Question
The question used in the VR interaction needed to be carefully selected utilizing
several criteria: resources necessary to answer it, length of time to answer, and definite
measurable answer. Because the investigator is not affiliated with the universities under
study, the selection of the reference question had to take into account the investigator
would not have access to each university’s proxy server and passwords needed to access
the university’s subscribed databases or rely on licensed software. The question was
researched to ensure the answer could be found using resources readily available on the
internet or in standard print sources.
29
The question must be answerable in a reasonable length of time (ten to twenty
minutes) and must have a definite measurable answer. During an interview with one VR
coordinator typical questions asked during VR reference were discussed. The majority of
the questions asked of this particular university’s VR service include: library logistics
(hours, locations, etc), assignment questions, directory-type questions, and social science
and humanities questions. (Sessoms, 2002)
Consequentially, the question is in two parts and has a three part answer, can be
answered in a reasonable length of time and can be answered only using the internet. The
question is: “When was the National Institute of Diabetes and Digestive and Kidney
Diseases (NIDDK) founded? What branch on the U.S. Public Health Service is it in, and
what year was the branch founded?” (Gollop & Tibbo, 2001) The resulting answers are
discussed in the Results of this paper.
Data Collection
Each site was randomly assigned a Site Number between one and ten. These
numbers are the only reference to the sites throughout this paper, keeping the identity of
the locations completely anonymous. This number is the only identifying information on
parts two through four of the data gathering forms. The data gathering forms include the
Site Demographics Form, Site Evaluation Form, VR Interaction Form, and VR
Evaluation Form (See Appendix A).
Site Demographics Form - The Site Demographics Form, the only page that
links the Site Number to the school name, is designed to collect the school name, web
addresses, the size of the undergraduate and graduate student body, the Carnegie
30
Classification, the Association of Research Libraries (ARL) library rank, and the name of
the library’s VR service.
Site Evaluation Form - The Site Evaluation Form evaluates the school’s library
website by the following Facets of Quality for Digital Reference Services: accessible,
publicize, private, clear response policy, and provides access to related information. The
sites are evaluated on the “Base” level of service for all facets:
• Facet 1: Accessible - The base level of quality for Accessible is described as
“Ensure that the service is reachable and navigable…” (VRD, 2000). Accessible
is evaluated by noting the location of links to VR reference, the visibility of those
links, the number of clicks required to access VR, availability and location of
quick links, the population being served and whether access is restricted to certain
users, Americans with Disabilities Act (ADA) accessibility, and internet
connection. As an additional form of evaluation of accessibility the investigator
used a dial-up modem connection to access the services.
• Facet 3: Clear Response Policy - The base level of Clear Response Policy is
described as “Create and adhere to a clear response policy” (VRD, 2000). Clear
Response Policy is evaluated by the presence of a VR reference response policy.
• Facet 8: Private - The base level of Private is described as “Deny public access to
user names, e-mail or mailing addresses, and questions unless there is written
notice prior to the information being made available…” (VRD, 2000). Private is
evaluated by the presence of a clear privacy policy and the location of that policy.
• Facet 10: Provides Access to Related Information - The base level of Provides
Access to Related Information is described as “Include basic resources on the
31
Web site to supplement the question-answering component” (VRD, 2000). This
is evaluated by the presence and location of supplementary resources such as
subject guides and FAQs.
• Facet 11: Publicize - The base level of Publicize is described as “Create and post
publicly a description of the purpose and practices of the service.” (VRD, 2000).
Publicize is evaluated by the availability of information about the VR service,
instructions for use, tips and Frequently Asked Questions (FAQ), location and
visibility of the hours of operation. A FAQ is a resource many libraries include
on their Web pages to address some of the questions most frequently asked of the
reference staff.
Virtual Reference Interaction Form - The VR Interaction Form contains the
script used in the VR interaction with the sites. This script included an introductory line,
the main question, the follow-up question, and a closing. While additional exchanges
were at times necessary during the reference interview, these exchanges were kept to a
minimum in order to maintain consistency between the sites. Spaces on the form allowed
for annotations during the VR interaction and for recording time markers.
Virtual Reference Evaluation Form - The VR Evaluation Form evaluates the
reference exchange by the base level Facets of Quality for Digital Reference Services
qualities of Prompt Turnaround, Instructive, and Interactive.
• Facet 2: Prompt Turnaround - The base level of Prompt Turnaround is
described as “Questions should be addressed as quickly as possible” (VRD,
2000). Prompt Turnaround is evaluated by the time taken for the first
response, total time for the interaction and general pace of subsequent
32
responses; the correctness of answers; ease of understanding; and
appropriateness of responses.
• Facet 4: Interactive - The base level of Interactive is described as “Encourage
sharing of important information…and question subject-area…” (VRD,
2000). Interactive is evaluated by the effectiveness of the reference interview,
and the human-touch of the reference interaction.
• Facet 5: Instructive - The base level of Instructive is described as “Offer
answers or pointers to information in responses to users. When unable to
provide an answer, provide the user with appropriate notification” (VRD,
2000). Instructive is evaluated by the authority of cited sources, follow-up
and avenues for further investigation are provided, and instruction and
resources are furnished while providing the answer.
Test Procedures
To conduct the test of the ten sites the investigator first established all the possible
correct answers to the VR session question, then logged on to all ten sites to establish the
VR reference hours and the time zone of the site. After establishing the time periods
most of the sites were available the investigator conducted all the sessions within a three
day period. At the beginning of each session the Site Demographics and the Site
Evaluation forms were filled out and any pertinent information was printed for future
reference. Three of the ten sites were only able to be evaluated using these first two
forms due to accessibility issues or due to the investigator being associated with the
university as an employee or as a student.
33
The VR Interaction form was filled out during the VR session, making note of the
resources used, the time of each exchange, and any additional noteworthy information.
The VR Evaluation form was filled out immediately after the session and reviewed once
the session transcript was received by e-mail and printed.
Additional protocols taken during the testing process include steps to ensure
investigator anonymity and notification of the sites that they were a part of the study. To
maintain investigator anonymity the following steps were taken: Logged in as an
undergraduate/off-campus student, used a non-university affiliated e-mail address, and
used an off-campus computer that has a non-university Internet Protocol (IP) address.
Within two months after the VR sessions were completed a follow-up message was sent
to the schools explaining the study and the unobtrusive research methods.
Data Analysis
Following the completion of the ten VR reference site investigations and seven
VR reference interactions the following steps were taken in the data analysis: each part of
the Facet was evaluated and given a scale score; all the scale scores for the facet were
averaged, resulting in the Facet score; totals were then created for each site. These totals
created the Site Evaluation Score and the Site Total Score. The scoring for the facets was
based on a sliding scale as follows: 0 = Facet not present, 1 = Facet fails to meet Base
Level, 3 = Facet meets Base Level, 5 = Facet exceeds Base Level, and NA = Not
Applicable.
The Results section of this paper includes each site’s results for each facet, a
general overview of the interaction, and a summary of the site. The Analysis section of
34
this paper includes a series of charts created to evaluate the Facets of Quality for Digital
Reference Services, evaluating sites on the Base Level of quality. The analysis reports
how well the sites meet the Facets of Quality and discusses generalizations about the state
of VR reference as concluded from the data gathered.
Results
Expected Answers
During the virtual reference interaction the question being asked of each site was
the same: “When was the National Institute of Diabetes and Digestive and Kidney
Diseases (NIDDK) founded? What branch of the U.S. Public Health Service is it in, and
what year was that branch founded?”(INLS 111, 2001). The anticipated answer to this
question is in three parts, these are NIDDK history, National Institutes of Health (NIH)
history, and United States Department of Health and Human Services (HHS) history.
The answers to these three parts are:
• NIDDK was founded in 1950 when President Truman signed an act establishing
National Institute of Arthritis and Metabolic Diseases. NIDDK is a part of NIH
(NIDDK, 2003). In 1985 the Health Research Extension Act of 1985, P.L. 99-
158, changed the institute name to the National Institute of Diabetes and
Digestive and Kidney Diseases.
• NIH was founded in 1887 and is a part of the US Department of HHS (NIH,
2003).
35
• The US Department of HHS has roots back to 1798; however it was not
established as a Cabinet-level Department until 1953 when it was called the
Department of Health, Education and Welfare. The name was changed in 1980 to
the Department of Health and Human Services (HHS, 2003).
Facet Scoring
The scoring process involved assigning a numeric value to the components of
each facet as described in the Methodology. The scores from all the facets were added up
to create the Site Evaluation Total score and the Site Total score. Three sites only have
the Site Evaluation Total component of the evaluation completed due to access
limitations; these sites do not have a Site Total score, only a Site Evaluation Total score.
Table 1 shows the Site Evaluation Total and Site Total for each site.
Table 1: Site Evaluation Total and Site Total scores by site number
SiteSite
Evaluation Total
Site Total
1 17 252 9 153 13 NA4 13 195 19 266 10 217 9 188 7 N9 15 2710 7 NA
A
When investigating the sites for the location of links and visibility of information
a general hierarchy of information revealed itself. The following pages will be referred
36
to and are defined here for consistency. General relationships between the pages are
represented in Figure 1 below.
• University Home Page – This is the main home page for the university.
• Library Home Page – This main page for the university libraries contains links to
the library resources and services. Ideally there is a link directly to the Reference
Help page.
• Reference Home Page – This page usually contains links to the Reference Help
page, subject guides, tutorials, ready reference links, FAQs, etc…
• Reference Help Page – Often called “Ask Us” this page contains information for
contacting reference help. This includes contact information for subject
specialists, desk hours, phone numbers, asynchronous reference service and
virtual reference service.
• Virtual reference (VR) Page – this is the page where the user connects to the
service. It usually contains usage tips, FAQ, and privacy policy about the VR
service.
37
FIGURE 1 – Typical relationship of links for access to library Virtual Reference service.
Asynchronous Digital Reference Page Reference Help Page
Library Home Page
Reference Home Page
Virtual Reference Page
University Home Page
The sites had a number of combinations of linkages between the Library Home
Page, the Reference Home Page, the Reference Help Page, the Asynchronous Digital
Reference Page, and the Virtual Reference Page. None of the University Home Pages
had a link to anything but the Library Home Page. This diagram represents the typical
hierarchy of links with a solid arrow ( ) and the additional arrangements that
were encountered with a dashed arrow ( ).
Individual Site Results
In addition to the general demographics, the results for each site will be reported
in relation to the Virtual Reference Desk (VRD) Facets of Quality for Digital Reference
Services (2000). These will be discussed in relation to whether it meets, exceeds, or fails
the base level of quality for each facet. Throughout the reporting and analysis of results
the library faculty or staff member facilitating the VR session will be referred to as a
38
librarian even though there is no way to confirm this is the case. None of the sites
indicated whether the VR sessions were staffed by professional librarians, library
students, or paraprofessionals.
Site 1 Results
Site 1 has approximately 40,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and ranked in the top 20
of ARL library ranking. This site received a total score of 25 with 27 being the highest
score received and a site evaluation score of 17 with 19 being the highest score received.
Facet 1 - Accessible (score = 3) – The Web site for Site 1 includes a link to the
synchronous reference service on the main library title bar, making it very visible and
easy to access. It takes three clicks to reach the service from the university home page.
This service is focused on the immediate university community, but anyone can use this
service; access is not restricted. There were no problems encountered with connection to
the service. ADA accessibility issues are not addressed.
Facet 2 – Prompt Turnaround (score = 3) – The whole interaction lasted 18
minutes; the first response came within one minute; subsequent responses were timely.
The librarian cited the NIDDK history and mission page, copied and pasted the NIH
history paragraph, but did not address the department. This site did not push the web
pages, but instead copied and pasted them into the VR session. Total time = 18 minutes.
Facet 3 – Clear Response Policy (score = 3) – The response policy is a part of
the definition of the virtual reference service.
39
Facet 4 – Interactive (score = 2) – The reference interview was effective with the
librarian asking questions such as “Do you need more academic information about these
agencies in the form of books or articles?” The lack of “pushing” Web pages and the
lack of the librarian’s name in the interaction caused a less than personal feeling to the
session; however the content of the conversation was very friendly and pleasant.
Facet 5 – Instructive (score =3) – Librarian used authoritative Web sites and
gave the addresses, and offered to find print sources; however there was no real
instructive element to the interaction.
Facet 8 – Private (score = 3) – Links to the privacy statement are on both the
Reference Help Page and the VR Page from a section about the VR service to the
confidentiality/privacy policy.
Facet 10 – Provides Access to Related Information (score = 5) – Site contains
links to FAQ type information and research guides from the Reference Help Page.
Facet 11 – Publicize (score = 3) – The VR Page contains basic information about
the service including information about possible technical difficulties. Hours of
operation are posted on the Reference Help Page and the VR Page.
Site 2 Results
Site 2 has approximately 20,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Intensive, and is not ranked by the
ARL library ranking. This site received a total score of 15 with 27 being the highest
score received and a site evaluation score of 9 with 19 being the highest score received.
40
Facet 1 – Accessible (score = 4) – Links to the VR service are located on the
Library Home Page and on the electronic resources page in the form of a pull down menu
and a large button. It takes two clicks to access VR from the University Home Page.
University affiliates are the primary target group; however anyone can access this
service. ADA accessibility issues are not addressed. There were no internet connection
problems.
Facet 2 – Prompt Turnaround (score = 2) – It took this site three minutes to
respond to the first log-on type statement; however the rest of the session went at a good
pace. The total time = 22 minutes. The NIDDK site was consulted and cited, and the
uniform resource locator (URL) was sent; gave NIDDK as being a part of NIH but did
not include date; did not look any further for HHS information. A URL is a Web site’s
unique internet protocol (IP) address.
Facet 3 – Clear Response Policy (score = 0) – There is no mention of a response
policy.
Facet 4 – Interactive (score = 2) – The librarian asked good questions; however
the follow-up was poor and the whole question was not addressed. The conversational
tone was friendly.
Facet 5 – Instructive (score =2) – There was no instruction. The NIDDK site is
the only site given or referenced, no alternative sources were given and there was no
follow-up. There was actually misleading information given, one example includes the
librarian stating that the NIH is a branch of the U.S. Government.
Facet 8 – Private (score = 0) – There is no privacy policy.
41
Facet 10 – Provides Access to Related Information (score = 1) – Links to
subject specialists and subject guides are included on the VR Page.
Facet 11 – Publicize (score = 4) – The hours of operation are located on the VR
Page and on a page designed for first-time users. Hardware and software requirements
are addressed as well as a trouble shooting tip.
Site 3 Results
Site 3 has approximately 30,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and is ranked in the top
50 by the ARL library ranking. This site did not receive a total score because the VR
interaction was not completed. The site evaluation score is 13 with 19 being the highest
score received.
Facet 1 – Accessible (score = 3) – Links to the Reference Help Page are located
on the side menu on the Library Home Page and in the top menu bar on all other pages.
The only link to VR is from the Reference Help Page, this is a text link. There is no quick
link from the Library Home Page and it takes three clicks to access the service.
University affiliates are the primary target group; however anyone can access this
service. ADA accessibility issues are not addressed.
Facet 2 – Prompt Turnaround (score = NA)
Facet 3 – Clear Response Policy (score = 1) – The only reference to a response
policy is in the phrase “chat now with a librarian online.”
Facet 4 – Interactive (score = NA)
42
Facet 5 – Instructive (score = NA)
Facet 8 – Private (score = 3) – The privacy policy is available on the Reference
Help Page.
Facet 10 – Provides Access to Related Information (score = 3) – There is a
FAQ and a guide to what types of questions are appropriate for digital reference.
Facet 11 – Publicize (score = 3) – There is no reference to instructions or tips for
use. The service hours are posted on the Reference Help Page in the form of a pop-up
window.
Site 4 Results
Site 4 has approximately 25,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Intensive, and is not ranked in the
ARL library ranking. This site received a total score of 19 with 27 being the highest
score received and a site evaluation score of 13 with 19 being the highest score received.
Facet 1 – Accessible (score = 3) – Links to the Reference Help Page are located
on the Library Home Page and the electronic resources page. These links are highly
visible; however even with a quick link from the Library Home Page it takes a total of
four clicks to finally reach the VR service from the University Home Page. University
affiliates are the primary target group; however anyone can access this service. ADA
accessibility issues are not addressed. There were no internet connection problems.
Facet 2 – Prompt Turnaround (score = 2) – The first response took one minute;
at one point there was a four minute gap in the exchange. During the session the NIDDK
43
mission page was given, and the HHS home page was given. There was no mention of
NIH and no page for HHS history. Through the whole session the librarian never gave
any dates, just gave a page so that “if you take a look at their mission/history page, I think
you’ll find some information.” The librarian also assumed the patron was comfortable
finding and interpreting the information on the web pages. Total time = 21 minutes.
Facet 3 – Clear Response Policy (score = 3) – The only reference to a response
policy is found on the Reference Help Page where the service is described as “real-time”.
Facet 4 – Interactive (score = 1) – The initial reference interview was barely
adequate with one clarifying question and one “Have you tried their web site yet?”
question. The tone was not very friendly.
Facet 5 – Instructive (score = 3) – Used the authoritative Web pages for NIDDK
and HHS. Was not very instructive and left the patron to find the answers on their own.
Librarian did not seem to want to investigate very much for the patron.
Facet 8 – Private (score = 3) – This site has a very thorough privacy policy with
a link on the VR Page.
Facet 10 – Provides Access to Related Information (score = 1) – Information
included from the VR Page includes library hours, the library catalog, electronic
resources and interlibrary loan. There was no link to FAQs or other guides or tutorials.
Facet 11 – Publicize (score = 3) – The VR Page contains thorough information
on the service including who can use it, how to use it, technical requirements, and tips for
use. The hours of operation are only located on the VR Page.
44
Site 5 Results
Site 5 has approximately 50,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and ranked in the top 50
of the ARL library ranking. This site received a total score of 26 with 27 being the
highest score received and a site evaluation score of 19 with 19 being the highest score
received.
Facet 1 – Accessible (score = 4) – A very visible button linking the user to the
Reference Help Page is located on many web pages and a text link is located on
additional pages. Despite all of the buttons and links, it still requires four clicks to access
the VR Page from the University Home Page. University affiliates are the primary target
group; however anyone can access this service. ADA accessibility issues are not
addressed. There were no internet connection problems.
Facet 2 – Prompt Turnaround (score = 3) – First response took two minutes.
Total time = 26 minutes. The session moved at a good pace with check-in type messages
sent periodically to keep the patron from feeling abandoned. This site did not find the
NIDDK mission and history page on the NIDDK site, tried MEDLINE and Google and
was asking the investigator for email contact information when the follow-up question
was sent by the investigator. The answer that was given stated that the NIH was a branch
of the United States government and the dates cited were incorrect. To quote the
librarian: “I had to call our Govt. Documents Dept. They have a guide called Andriot’s
Guide to U.S Govt. publications. It says that the [NIDDK] was est in 1985. The [NIH]
was est. in 1970.”
45
Facet 3 – Clear Response Policy (score = 3) – The only reference to a response
policy is found on the Reference Help Page where the service is described as “live.”
Facet 4 – Interactive (score = 2) – The librarian seemed to be in a hurry,
somewhat impatient and therefore missed the answers and did not acknowledge the
difference between a US Department and an institute. The tone of the interaction was
kind and conversational.
Facet 5 – Instructive (score = 2) – The full citation was not given for the
“Andriot’s Guide to U.S. Govt publications.” The web sites that were used were
authoritative. The only instruction given is the web addresses.
Facet 8 – Private (score = 3) – The VR Page contains a link to the privacy policy.
Facet 10 – Provides Access to Related Information (score = 5) – The visible
options from the Reference Help or the VR Pages are references to other Reference Help
services and a tool bar link to “Help.” Help contains subject guides, tutorials, research
guides, and an online instructional video.
Facet 11 – Publicize (score = 4) – The VR Page contains information about the
technical requirements and tips for possible problems. The hours of operation are located
on both the Reference Help Page and the VR Page.
Site 6 Results
Site 6 has approximately 20,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and ranked in the
bottom 50 of the ARL library ranking. This site received a total score of 21 with 27
46
being the highest score received and a site evaluation score of 10 with 19 being the
highest score received.
Facet 1 – Accessible (score = 2) – Links to the Reference Help Page are available
on the home page as a roll over, and on the standard library tool bar. It takes five clicks
to access the VR service; there is not a direct link from the library home page. University
affiliates are the primary target group; however anyone can access this service. ADA
accessibility issues are not addressed. There were no internet connection problems.
Facet 2 – Prompt Turnaround (score = 3) – The first response took two
minutes; the total time = 22 minutes. The session moved at an adequate pace. The
NIDDK mission page was given; NIH date was given, but did not give the HHS page.
Librarian did acknowledge that the NIH is a part of HHS. The session was easy to
understand and appropriate for an undergraduate level of expertice.
Facet 3 – Clear Response Policy (score = 0) – There is no mention of a response
policy or time requirement.
Facet 4 – Interactive (score = 4) – The session had an appropriate personal touch
and conversational tone with frequent updates on the process of searching. Co-browsing
was very effective.
Facet 5 – Instructive (score = 4) – The web pages used were authoritative. The
librarian inquired about the user’s knowledge of appropriate citation techniques and sent
a web resource for user reference. Follow-up questions were appropriate; however there
was no follow-up on the HHS date.
47
Facet 8 – Private (score = 3) – There is a link to the privacy policy from the VR
Page.
Facet 10 – Provides Access to Related Information (score = 3) – The reference
Help Page contains links to subject specialists and subject guides.
Facet 11 – Publicize (score = 2) – The hours of operation are only located on the
VR Page. That page also contains tips for use and trouble shooting.
Site 7 Results
Site 7 has approximately 20,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Intensive, and is not ranked by the
ARL library ranking. This site received a total score of 18 with 27 being the highest
score received and a site evaluation score of 9 with 19 being the highest score received.
Facet 1 – Accessible (score = 4) – links to the Reference Help Page are located
on the Library Home Page and on most other library pages in the menu bar. The number
of clicks to access the VR service from the University home page is five. This includes
the quick link found on the Library Home Page. University affiliates are the primary
target group; however anyone can access this service. ADA accessibility issues are
addressed through providing a phone number for calling the reference department. There
were no internet connection problems.
Facet 2 – Prompt Turnaround (score = 4) – The first response took two
minutes; the total time = 32 minutes. This time seemed quick due to the librarian’s
messages informing the user that certain activities would take a few minutes to complete.
48
This type of communication is very helpful and appropriate to the undergraduate level.
The answer for NIDDK was found on the web site and the answer for HHS was found in
the U.S. Government Manual. There was no mention of the NIH.
Facet 3 – Clear Response Policy (score = 3) – A statement on the Reference
Help Page and the VR Page explains VR as occurring in “real time.”
Facet 4 – Interactive (score = 3) – The reference interview was very effective;
utilizing clarification and follow-up questions appropriately. The interaction was very
conversational; casual yet effective.
Facet 5 – Instructive (score = 2) – There was little instruction. The web pages
were sent to the user for further investigation; however there was never a full citation
given for the U.S. Government Manual.
Facet 8 – Private (score = 0) – No privacy policy is posted.
Facet 10 – Provides Access to Related Information (score = 1) – The resources
that are available from the VR Page include resources to help users find articles, books,
and internet resources, and a series of quick reference resources.
Facet 11 – Publicize (score = 1) – Tips for use are only available after the user
logs on to the service. The hours of operation are only posted on the VR Page.
Site 8 Results
Site 8 has approximately 25,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and is ranked in the top
20 of the ARL library ranking. This site did not receive a total score due to being an
49
affiliate of the researcher. The site evaluation score is 7 with 19 being the highest score
received.
Facet 1 – Accessible (score = 1) – The only access to this service is through the
Reference Help Page and this link is not very visible. The total number of links to access
VR Service is four. University affiliates are the primary target group; however anyone
can access this service. ADA accessibility issues are not addressed.
Facet 2 – Prompt Turnaround (score = NA)
Facet 3 – Clear Response Policy (score = 3) – The VR Page contains a statement
explaining the service and the method of response.
Facet 4 – Interactive (score = NA)
Facet 5 – Instructive (score = NA)
Facet 8 – Private (score = 0) – No privacy policy is provided.
Facet 10 – Provides Access to Related Information (score = 0) – There are no
links to additional resources from the VR Page.
Facet 11 – Publicize (score = 3) – Possible technical difficulties are explained.
Hours of operation are available on the VR Page and the Reference Help Page.
Site 9 Results
Site 9 has approximately 40,000 graduate and undergraduate students, is classified
by the Carnegie Classification as Doctoral/Research – Extensive, and ranked in the top 20
by the ARL library ranking. The site received a total score of 27 with 27 being the
50
highest score received and a site evaluation score of 15 with 19 being the highest score
received.
Facet 1 – Accessible (score = 3) – Links to the Reference Help Page are on all
library pages with a large graphic button or a text link. It takes seven clicks to get to the
VR Page form the University Home Page. University affiliates are the primary target
group; however anyone can access this service. ADA accessibility issues are not
addressed. There were no internet connection problems.
Facet 2 – Prompt Turnaround (score = 4) – The first response took one minute
and the total time = 17 minutes. The whole session proceeded at a good pace. All three
sites with the history information were located and sent to the user. During co-browsing,
a big red cursor shows the user where the librarian is clicking, this is helpful so the user
can follow the librarian’s navigation. Pages are pushed at appropriate times.
Facet 3 – Clear Response Policy (score = 3) – The response policy consists of a
statement about the service.
Facet 4 – Interactive (score = 4) – The reference interview is highly effective
with precise clarification and follow-up questions. The conversational tone is not very
friendly; however the interaction is appropriate and professional.
Facet 5 – Instructive (score = 4) – Official web sites are used and the addresses
are provided in the transcript. Instructional information about using a web address to
discover more about a site is explained; “the web address is http://niddk.nih.gov/, which
implies that it is part of the NIH (National Institutes of Health). Maybe we can find more
info there.”
51
Facet 8 – Private (score = 3) – The privacy policy is located on the VR Page.
Facet 10 – Provides Access to Related Information (score = 3) – A FAQ is
located on the Reference Help Page.
Facet 11 – Publicize (score = 3) – The information about the service is very well
organized and easy to navigate. This information covers technical problems, use tips, and
technical requirements. The hours of operation are only available on the VR Page.
Site 10 Results
Site 10 has approximately 20,000 graduate and undergraduate students, is
classified by the Carnegie Classification as Doctoral/Research – Extensive, and is ranked
in the bottom 50 by the ARL library ranking. This site did not receive a total score due to
highly restricted access. The site evaluation score is 7 with 19 being the highest score
received.
Facet 1 – Accessible (score = 2) – Links to the service are located on the bottom
tool bar of each page and on the Library Home Page. This text link is difficult to locate
due to having to scroll to the bottom of the page. It only takes two clicks from the
University Home Page to reach the VR Page due to the quick link form the Library Home
Page. University affiliates are the only group that can access this service and they must
have their proxy server configured to enter the VR page. ADA accessibility issues are
not addressed.
Facet 2 – Prompt Turnaround (score = NA)
52
Facet 3 – Clear Response Policy (score = 3) – The service is referred to as
“interact live with a reference librarian”.
Facet 4 – Interactive (score = NA)
Facet 5 – Instructive (score = NA)
Facet 8 – Private (score = 0) – There is no privacy policy available.
Facet 10 – Provides Access to Related Information (score = 0) – No resources
are available from the VR Page or the Reference Help Page.
Facet 11 – Publicize (score = 1) – Technical requirements are available, but no
hours of operation are available.
Analysis
Given the wide variety of data gathered, there are numerous options for analysis.
This analysis focuses on each of the eight facets studied, giving an example of each with
the data gathered; and, as applicable, a comparison to other studies either supporting or
refuting this data. The hours of operation are given a separate section due to the
extensive amount of data involved. The researcher has elected to examine these areas
because the hours of operation are directly related to library staffing resources and the
accessibility of the service by users, and the general facet results and trends reveal
strengths and weaknesses in the quality of virtual reference being offered. The analysis
will further discuss the limitations of the study and implications for future investigations.
53
Hours of operation
In Sloan’s (2001) “report on the Ready for Reference project,” he discusses the
results of a study on a VR service that provides reference service 24 hours a day, seven
days a week. Three of the resulting points focus on an analysis of when the service was
utilized. This data is broken down by time of day, days of the week and months of the
year. The analysis of the hours of operation for this study will utilize Sloan’s findings as
a basis for comparison for the VR service hours offered each day and each week.
Table 2 shows the hours of operation of each site on a weekly calendar. This
visually oriented method of displaying the data makes the patterns more readily apparent.
The data from Table 2 is further represented by Table 3, Table 4, and Table 5.
54
Table 2: Virtual Reference Hours of Operation by Site Number
Monday Tuesday Wednesday Thursday Friday Saturday Sunday Totals
8:00 AM 0
9:00 AM 11
10:00 AM 25
11:00 AM 25
12:00 PM 20
1:00 PM 38
2:00 PM 36
3:00 PM 40
4:00 PM 26
5:00 PM 13
6:00 PM 16
7:00 PM 20
8:00 PM 15
9:00 PM 8
10:00 PM 4
11:00 PM 4
12:00 AM 0
Totals 68 63 63 54 35 4 14 301
1
2 3 4
5
6
7 8
9
3
1
3
9
1 2
3 4
5
6
8
1 2
3 4
5
6
9
8
1 2
3 4
5
6
7
9
8
21
3 4
5
6
9
8
Note: Times reflect the site’s own time zone. 54
55
As Table 2 and Table 3 reflect, Site 7 is at the low end of the hours of operation
with only twelve hours per week over only two days, while Site 3 is at the high end of the
range with 68 hours per week over seven days.
Table 3: The total number of hours and days per week by VR site number
Site Hours Days1 33 52 37 53 68 74 27 55 20 56 35 57 12 28 27 59 39 510 NA NA
In Helman’s (2001) article analyzing when users think VR should be available
and when users would use VR, the “users …expressed an interest in seeing this service
available from 9 am to 9 pm. When users were asked when they would be most likely to
use AskUs!-Live, most users (38%) preferred 12 pm to 3 pm, while 3 pm to 6 pm was a
close second (29%).” (p. 87) The findings of this study are reflected in Table 4, with
most of the sites’ hours of operation being available from 12 pm to 3 pm. The second
and third most available hours of operation are 9 am to 12 pm and 4 pm to 8 pm
respectively.
Conversely, Sloan (2001) discussed VR hours of user activity by stating:
It is interesting to note that less than half (43.4 percent) of all activity took place between ‘typical’ business hours of 8a.m. and 5p.m. Even more interestingly, there was more activity between 5p.m. and 1a.m. (43.7 percent of total) than between 8a.m. and 5p.m. Typical morning business hours (8a.m. to 12 noon) account for only 10.5 percent of the total. And the early hours of the morning (1a.m. to 6a.m.) do not generate much activity at all (only 18 sessions in 22
56
weeks, or less than 3 percent of total activity). This might suggest a re-evaluation of the goal to offer reference services 24 hours a day. (p. 15)
Table 4: The total number of VR hours per week by hour of the day
0
5
10
15
20
25
30
35
40
8:00 AM
9:00 AM
10:00 AM
11:00 AM
12:00 PM
1:00 PM
2:00 PM
3:00 PM
4:00 PM
5:00 PM
6:00 PM
7:00 PM
8:00 PM
9:00 PM
10:00 PM
11:00 PM
12:00 AM
Sloan (2001) goes on to discuss the days of the week VR is used by stating:
Sessions appear evenly distributed from Monday through Thursday. Wednesdays turned out to be the busiest day of all. There is a clear division between the ‘major’ weekdays and the weekend (Friday, Saturday, and Sunday). Approximately 74 percent of all sessions occurred from Monday through Thursday, with only 26 percent occurring from Friday through Sunday. If only Saturday and Sunday are counted as weekend days, the weekend share drops considerably more, with only 14 percent of total sessions. This might suggest that staffing requirements are less on the weekends, and that the goal of offering a live online service seven days a week might be revisited. (p. 15)
Using Table 5 as a comparison it is evident that the sites studied offer VR on days
consistent with Sloan’s findings. Monday through Thursday constitute eighty-two
57
percent (82%) of the hours of operation among the ten sites, with Friday through Sunday
comprising the remaining eighteen percent (18%).
Table 5: Total number of VR hours per day of the week
0
10
20
30
40
50
60
70
Monday (68)
Tuesday (63)
Wednesday (63)
Thursday (54)
Friday (35)
Saturday (4)
Sunday (14)
Facet Results
One of the benefits of evaluating digital reference service is the ability of one
person to evaluate the service unobtrusively; by having the perspective of only one
investigator there is greater consistency. This section will report the findings from each
of the Facets of Quality for Digital Reference that was investigated. Each facet could
make, at the highest, a score of 50 with the Facet Total “base level” score being 30. The
results show that VR service needs to improve in all facets evaluated in order to meet the
“base level” of the Facets of Quality for Digital Reference.
58
3
3
Table 6: Facet scores by site number and Facet number
FacetSite
1 2 3 4 5 8 10 11
1 3 3 3 2 3 3 5 32 4 2 0 2 2 0 1 43 3 NA 1 NA NA 3 34 3 2 3 1 3 3 1 35 4 3 3 2 2 3 5 46 2 3 0 4 4 3 3 27 4 4 3 3 2 0 1 18 1 NA 3 NA NA 0 09 3 4 3 4 4 3 3 310 2 NA 3 NA NA 0 0 1
Facet Total 29 20 20 18 20 18 22 27
Facet 1 – Accessible
An evaluation of the scores for Facet 1 shows that this facet most closely meets
the “base level” of standard with a total score of 29. The area of this facet which needs
the most improvement is ADA issues, these are only addressed by one site (Site 7) and
this is to give out a phone number for directly contacting the reference department, not
online instructions about how to make a VR session more accessible. The number of
clicks required to access VR service as an additional indicator of accessibility revealed
that four sites required two to three clicks, five required four to five clicks, and one
required six to seven clicks to access VR. None of the services had a quick link from the
university home page, six had a quick link from the library home page. These links are
most often text links (10), and/or a graphic or button (4). The links are easily visible and
do not require scrolling at seven sites.
59
Facet 2 – Prompt Turnaround
Prompt Turnaround received a total score of 20, not meeting the Base Level of the
facet. One way this study looked at Prompt Turnaround was the length of time it took for
a librarian to answer the VR log-in. According to Sloane (2001) “It was important to log
the length of time users waited in a queue. The results were that 63 percent of users
waited fewer that 30 seconds before being contacted by a librarian, and 73.5 percent were
contacted in one minute or less. This indicates that, for the sample studied, librarians
were very prompt in responding to patron queries.” The wait time encountered during
this study was between one to three minutes for all sites (see Table 7).
Duration or total VR session time is another indicator of Prompt Turnaround. In
the study by Kubee, Ward, and Ma “the average time spent in a chat session was 9.8
minutes. The shortest completed transaction was 40 seconds and the longest was 58.50
minutes.” (2002, p. 33) Sloan’s 2001 study showed a similar variety in duration of
session, he states:
The duration of online reference sessions varied greatly for the sample studied, ranging from two minutes and four seconds (2:04) to 54 minutes and 26 seconds (54:26). The median session length was 13 minutes and 11 seconds (13:11). Only 12.2 percent of sessions lasted fewer than five minutes; 21.4 percent lasted from five to ten minutes; 25.5 percent lasted from ten to 15 minutes; 14.3 percent lasted from 15 to 20 minutes; and 26.5 percent lasted longer than 20 minutes. (p. 16)
As compared to these two studies the duration of the seven sessions ranged from
17 minutes to 32 minutes (see Table 7). This is a wide range considering all sites were
asked to answer the same questions.
60
Table 7: Wait time for first response and VR session duration by site number (in minutes)
Site First response Duration1 1 182 3 223 NA NA4 1 215 2 266 2 227 2 328 NA NA9 1 1710 NA NA
Prompt Turnaround (Facet 2) also included the correctness of answers. By
integrating this aspect of the evaluation into one of the facets, the focus on correct
answers is minimized. While this may seem to be a disadvantage, it actually allows
libraries whose focus is information literacy and integrating instruction into the reference
encounter to be evaluated fairly. The question asked of each site had three answer
components. Site 9 is the only site to answer all of these components correctly. Sites 1
and 6 answered two of the three components correctly, Sites 2, 4, and 7 only answered
one component correctly, and Site 5 did not answer any of the components correctly.
That translates to a response rate of only 48 %.
While the correctness of answers was not satisfactory, the sessions were easy to
understand with the responses worded at a level that was appropriate for undergraduate
students.
Facet 3 – Clear Response Policy
None of the sites have a Clear Response Policy posted, however they were given
credit if the service was described as “live,” or “chat in real-time with a librarian.” With
61
this concession the total score for this facet is 20. This omission was basically assuming
the user knew that VR is instantaneous. There is no discussion of the possibility of
having to wait for the librarian to log on or having to wait for the users ahead to finish
their session. Some software has the ability to notify the user that they are “in-line” for
service, however none of these sites indicate that is a possibility.
Facet 4 – Interactive
Interactive (Facet 4) received a total score of 18. This Facet evaluated the
effectiveness of the reference interview and the tone of the interaction. All Sites except
Site 5 utilized clarifying questions, only three sites asked follow-up questions at the end
of the session, and only two sites fully investigated the questions being asked. The VR
interactions all had a conversational/friendly tone except Site 4, which did not. The VR
conversation with Site 1 contained gaps in the conversation that evoked a feeling of
wondering if they were still there.
Facet 5 – Instructive
Instructive (Facet 5) scored a total of 20. The evaluation of this Facet was based
on the use of authoritative sources, the provision of avenues for further investigation, and
providing instruction while also providing the answer. One tool that facilitates this
process is the ability to “push” Web pages to the user and the ability to co-browse. Co-
browsing allows the user to see all the actions the librarian takes while using online
resources. Site 1 did not utilize these features, instead this Site would copy and paste
from Web pages into the VR text message portion; this posed a problem when the
amount of information was very large.
62
All of the Sites used authoritative Web sites when referring the user to the
answers, only two sites used print resources. For further investigation, the Sites provided
the URLs to the sites visited; only Site 6 offered the link to a resource for adequately
citing the resources user during the session. Site 2 was the only site that did not provide
any instruction throughout the session, four of the sites provided minimal instruction,
while two sites did a good job of instructing the user.
Facet 8 – Private
The Sites scored a total of 18 for the Private Facet. To evaluate this Facet each
site was thoroughly investigated to find a privacy policy about digital reference services.
Six of the sites have confidentiality/privacy statements available; of these sites five of
them are linked directly from the VR Page and one is linked from the Reference Help
Page.
Facet 10 – Provides Access to Related Information
For this Facet the total score was 22. This Facet was evaluated by what
supplementary resources are available from the Reference Help Page and the VR Page.
These resources include, but are not limited to, FAQs, tutorials, research guides, or
subject guides. Only one site did not have links to any resources, six sites had links from
the VR Page, and eight had links from the Digital Reference Page.
Facet 11 – Publicize
Publicize (Facet 11) had a total score of 27. To evaluate this facet the researcher
looked at the visibility of the hours of operation, and the availability of trouble-shooting
63
tips and tips for use of the service. Site 10 was so restricted that users have to log-in to
find out more information about the service and its hours and must use a valid university
identification to log-in. Eight sites posted the hours of operation on the VR Page; five
posted the hours of operation on the Reference Help Page, with four having the hours
posted in both locations. Five sites include technical requirements for using the service,
six sites post tips for solving or preventing technical problems, and three sites include tips
for effective VR interaction. Two sites do not have any instructions or tips for use
available.
Discussion
Due to the experimental nature of this study there are a number of limitations that
were revealed during the course of this project. These include VR collaborations,
conversation consistency, form design, software, and Facets that were not evaluated.
With an increasing number of libraries utilizing VR collaborations it was difficult
to find a critical mass of libraries with VR service fitting the parameters set for the study.
If this study is replicated in the future, the parameters should be changed to include more
Carnegie Classification levels and include VR services that use collaborations with
institutions included in the specified Carnegie Classification levels.
While trying to maintain consistency in the conversations between each site the
researcher did not have the ability to persist if the interaction with the librarian was on the
wrong track. Additionally, while trying to “act” like an undergraduate student there were
times when the researcher did not reword the questions to get the desired answer. Both
conversational points pose a potential problem for the evaluation of VR accuracy in the
64
future. How do you maintain optimal consistency between each site and avoid leading
the interaction? In two conversations the researcher accidentally used a word incorrectly,
while an effective reference interview would clarify what the user meant, the VR sessions
in this study did not make the distinction and therefore provided the wrong answer.
When unobtrusively evaluating reference services one limitation is that the
institution might be focusing on information literacy and user instruction through the
reference interaction, while the study is focusing on factual answers. One way this study
accommodates for this is by using the Facets of Quality to evaluate not only answer
correctness, but also the full web presence and full interaction of the VR service. Within
the digital environment it is possible to unobtrusively evaluate the whole user experience.
The high-speed pace of VR especially lends itself to the tendency of librarians to
use “… a Web site address [as]…a shortcut response intended to ’get rid of’ a patron,
with no follow-up and no suggested alternatives should the original Web address fail to
meet an information need.” (Dilevko, 2000a, p. 306) The standards for VR need to
address the phenomenon of relying heavily on the internet and not providing follow-up
opportunities or alternate sources especially print ones.
The data collection forms, while effective for this study, need to be revised.
Moving some of the evaluative measures from one Facet to another and rewording some
of the measures will improve the facet evaluation by having the measures more directly
related to the Facet. Integrating the accuracy of the answers into one of the facets takes
the focus away from this aspect of the service and looks at the service as a whole. This is
just a reminder of Tyckoson’s (1992) statement about satisfaction with library service not
65
being linked as much to the correct answer as it is to the overall quality of service.
Patrons may not get the answer they are looking for, but if they have a good experience
then they are likely to return.
Additional limitations include technical problems the library might have with the
VR software. (Tobin, 2002) These technical problems can include accidental disconnect,
and the inability to co-browse or push pages.
Several Facets of Quality are not evaluated in this study due to a lack of
information available about these Facets in a publicly accessible location. Future
versions of the Facets of Quality and VR service Web sites should strive to integrate this
data into their documentation about the VR service. These omitted facets include
Authoritative, Trained Experts, and Reviewed.
Authoritative (Facet 6) should be included in the VR documentation as well as
stated in the VR welcome message. Users that log-on could see “Hello, welcome to our
VR service, Person A, a reference librarian (information specialist, graduate student, etc)
will be with you shortly.” By communicating the expertise of the VR facilitator the user
is more informed. The VR documentation can further explain what each “expertise”
statement means.
Trained Experts (Facet 7) can also be included in the VR documentation. Again,
by fully informing the user about the VR service he/she can be a more critical user of the
information provided during the VR session. This is especially important when trying to
model information literacy behaviors to the user community.
66
Reviewed (Facet 9); VR sites and services can regularly conduct unobtrusive
studies on themselves (not just transcript review). The feedback from regular evaluation
keeps the service providers from becoming stagnant in their VR skills and encourages a
high quality of service. The results of such unobtrusive evaluations can be integrated into
the VR documentation as well.
Future Investigation
The investigator of this study believes this study is another step forward in the
studies conducted on the effectiveness of VR. Some of the other areas for further
investigation include:
• Further investigations of VR utilizing the Virtual Reference Desk’s Facets of Quality
for Digital Reference.
• A comparison of the quality of answers and time for response between asynchronous
reference and VR.
• How is library staff trained for VR? What is covered? Are they provided further
training/development? How is the adaptation of the reference interview in digital
reference addressed?
• How do different large academic libraries handle the multiple forms of reference?
What models of reference service are utilized or adapted to integrate digital reference
services? How is digital reference facilitated and/or staffed? Is it done from one
location or within each librarian’s office? What are the successes and frustrations
with each model?
67
Conclusion
The virtual reference method of reference service delivery is still new and
exciting; this newness provides ample opportunities for experimentation. By using
unobtrusive methods to investigate a number of sites and how they utilize VR, the
investigator was able to look at the trends in this reference methods and compare them to
what is considered the standard for digital reference services. The findings in this study
include the implications for VR standardization utilizing the Virtual Reference Desk’s
Facets of Quality for Digital Reference.
The one area that VR seems to have achieved some consistency is accessibility as
demonstrated through the hours of operation. The Sites in this study reflect the hours that
users have requested and used in studies such as Sloan (2001) and Helman (2001). The
need for further standardization in VR is evident from the results of this study. When
libraries adapt and utilize the Facets of Quality for Digital Reference to achieve
standardization in VR service they will further the usefulness of that document,
expanding the focus from being primarily a tool for asynchronous digital reference to
including virtual reference service. The current version of the Virtual Reference Desk
document, however, does need to be revised to better fit the specialized capabilities and
limitations of VR service.
68
Bibliography
Bowman, V. (2002). The virtual librarian and the electronic reference interview. Internet
Reference Services Quarterly, 7 (3), 3-14.
Boyer, J. (2001). Virtual reference at North Carolina State: The first one hundred days.
Information Technology and Libraries, 20 (3), 122-128.
Broughton, K.M. (2001). Our experiment in online, real-time reference at Bowling Green
State University. Computers in Libraries, 21 (4), 26-31.
Bushallow-Wilbur, L., DeVinney, W., & Whitcomb, F. (1996). Electronic mail reference
service: A study. RQ, 35 (3), 359-371.
Carnegie Foundation for the Advancement of Teaching (Carnegie) (2000). Carnegie
Classification of Institutions of Higher Education, 2000 Edition. Electronic datafile,
second revision. 2002. Retrieved November 8, 2002, from
http://www.carnegiefoundation.org/Classification/CIHE2000/
Childers, T. (1990). Scouting the perimeters of unobtrusive study of reference. In B.
Allen (Ed.), Evaluation of Public Services Personnel (pp. 27-42). Urbana-
Champaign, IL: University of Illinois.
Crowley, T. (1985). Half-right reference: Is it true? RQ 25, 59-68.
Dewdney, P. & Ross, C.S. (1994). Flying a light aircraft: Reference service evaluation
from a user’s viewpoint. RQ 34, 217-230.
Dilevko, J. (2000a). ‘My mother can’t quite understand why I decided to go to library
school’: What patrons say about library staff when asking government documents
69
reference questions at depositing libraries. Journal of Government Information, 27,
3, 299-323.
Dilevko, J. (2000b). Unobtrusive evaluation of reference service and individual
responsibility: the Canadian experience. Westport, CT: Ablex Publishing
Corporation.
Foley, M. (2002). Instant messaging in an academic library: A case study. College &
Research Libraries, 63, 1, 36-45.
Gollop, C. & Tibbo, H. (INLS) (2001). Course Materials. INLS 111: Information
Resources and Services I. School of Information and Library Science, University of
North Carolina at Chapel Hill.
Helman, D. L. (2001). Bringing the human touch to digital library services. Science and
Technology Libraries, 20, 2/3, 83-96.
Hernon, P., & McClure, C. (1986). Unobtrusive reference testing: The 55 percent rule.
Library Journal, 111, 7, 37-41.
Hernon, P. & McClure, C. (1987). Library reference service: An unrecognized crisis – A
symposium. Journal of Academic Librarianship, 13, 2, 69-71.
Janes, J. (2002). Digital reference: Reference librarians’ experiences and attitudes.
Journal of the American Society for Information Science and Technology, 53, 7, 549-
566.
Janes, J., Carter, D. S., & Memmott, P. (1999). Digital reference services in academic
libraries. Reference & User Services Quarterly, 39, 2, 145-150.
Janes, J., & McClure, C. R. (1999). The Web as a reference tool: comparisons with
traditional sources. Public Libraries, 38, 1, 30-33+.
70
Johnson, C. M. (2002). Online chat reference: The awareness of, use of, interest in, and
marketing of this new reference service technology. A Master’s Paper for the M.S.
in L.S. degree. Advisor: Barbara M. Wildemouth. School of Information and Library
Science, The University of North Carolina at Chapel Hill.
Kibbee, J., Ward, D., & Ma, W. (2002). Virtual service, real data: results of a pilot study.
Reference services review, 30, 1, 25-36.
Lankes, R. D., Shostack, P. (2002). The necessity of real-time: fact and fiction in digital
reference systems. Reference& User Services Quarterly, 41, 350-355.
Maxwell, N. K. (2002). Establishing and maintaining live online reference service.
Library Technology Reports, 38, 4, 1-78.
Missingham, R. (2000). Virtual reference desk conference, October 2000, Seattle.
Australian Academic & Research Libraries, 31, 4, 208-210.
National Institute of Diabetes & Digestive & Kidney Diseases (NIDDK) (2003).
Welcome: Mission and history. Retrieved February 9, 2003, from
http://www.niddk.nih.gov/welcome/mission.html.
National Institutes of Health (NIH) (2003). Questions and Answers about NIH. Retrieved
February 9, 2003, from http://www.nih.gov/about/Faqs.htm.
Norlin, E. (2000). Reference evaluation: A three-step approach – surveys, unobtrusive
observations, and focus groups. College and Research Libraries, 61, 6, 546-553.
Peters, T. A. (2002). E-reference: how consortia add value. Journal of Academic
Librarianship, 28, 4, 248-150.
Qayyum, M.A. (2002). Internet reference services and the reference desk: Does the
nature of a user’s query really change? Internet Reference Services Quarterly, 7, 3,
15-22.
71
Riggs, C. (2002). Virtual reference services…what, why and how? A report on a
teleconference. Library Hi Tech News, 19, 4, 7-9.
Ross, C. S., Dewdney, P. (1998). Negative closure: strategies and counter-strategies in
the reference transaction. Reference & User Services Quarterly, 38, 2, 151-163.
Ross, C.S. & Nilsen, K. (2000). Has the Internet changed anything in reference? The
library visit study, Phase 2. Reference and User Services Quarterly, 40, 2, 147-155.
Ruppel, M., & Fagan, J. C. (2002). Instant messaging reference: Users’ evaluation of
library chat. Reference Services Review, 30, 3, 183-197.
Sessoms, P. (2003). Personal interview, on November 18, 2003.
Sloan, B. (2001). Reference service in the digital library: a report on the Ready for
Reference project. Library Hi Tech News, 18, 10, 14-19.
Slaon, B (2003). The Bernie Sloan Site. Retrieved January 25, 2003, from
http://alexia.lis.uiuc.edu/~b-sloan/bernie.htm.
Taher, M. (2002a). The reference interview through asynchronous e-mail and
synchronous interactive reference: Does it save the time of the interviewee? Internet
Reference Services Quarterly, 7, 3, 23-34.
Taher, M. (2002b). Real-time (synchronous interactive) reference interview: A select
bibliography. Internet Reference Services Quarterly, 7, 3, 35-41.
Teaching Librarian (2003). Index of Chat Reference Services. Retrieved January 25,
2003, from http://pages.prodigy.net/tabol/chatsoftware.htm.
Tennant, R. (2003). Revisiting Digital Reference. Library Journal, 128, 1, 38, 2p.
Retrieved February 2, 2003, from EBSCOHost.
Tenopir, C. & Ennis, L.A. (2002). A decade of digital reference: 1991-2001. Reference&
User Services Quarterly, 41, 3, 264-273.
72
Tobin, C. (2002). Personal interview, on November 19, 2002.
Tunender, H. (2002). Digital reference: trends, techniques and changes. Library Hi Tech
News, 19, 4, 5-6.
Tyckoson, D.A. (1992). Wrong questions, wrong answers: Behavioral vs. factual
evaluation of reference service. In S. G. Blandy, L.M. Martin, & M.L. Strife (Eds.),
Assessment and Accountability in Reference Work (pp. 151-173). New York: The
Haworth Press, Inc.
United States Department of Health & Human Services (HHS) (2003). Historical
Highlights. Retrieved February 9, 2003, from
http://www.hhs.gov/about/hhshist.html.
University of Iowa (LiveRef) (2003). LiveRef Registry. Retrieved January 25, 2003, from
www.public.iastate.edu/~CYBERSTACKS/LiveRef.htm.
Virtual Reference Desk Project (VRD) (2000). Facets of Quality for Digital Reference
Services. Version 4, October 2000. Retrieved February 2, 2003, from
http://www.vrd.org/facets-10-00.shtml.
White, M. D. (1999). Analyzing electronic question/answer services: framework and
evaluations of selected services (CLIS Technical Report No. 99-02).
White, M. D. (2001). Diffusion of an innovation: digital reference service in Carnegie
Foundation master’s (comprehensive) academic institutions libraries. Journal of
Academic Librarianship, 27, 3, 173-187.
Whitlatch, J.B. (1989). Unobtrusive studies and the quality of academic library reference
services. College and Research Libraries, 50, 2, 181-194.
Whitlatch, J. B. (2000). Evaluating reference services: a practical guide. Chicago:
American Library Association.
73
Whitlatch, J. B. (2001). Evaluating reference services in the electronic age. Library
Trends, 50, 207-217.
74
Appendix
Site Demographics Form
Site Evaluation Form
Virtual Reference Interaction Form
Virtual Reference Evaluation Form
75
Site Number
Site Demographics Form
School Name: URLs: (School)
(Library)
(VR)
Number of students: (Undergraduate) (Graduate)
Carnegie Classification:
ARL Library ranking:
Name/title of Library’s VR service:
76
Site Number
Site Evaluation Form
Facet 1: Accessible - “Base: Ensure that the service is reachable and navigable…” Beginning at the university home page and progressing toward the VR reference service.
Location of VR links: Visibility of VR links: (Is it necessary to scroll to find a link? Is it a text link or a graphic/button?) Number of clicks to access VR: Quick link from home page? From library home? Population being served (who can use it): Restricted access? How is that determined? ADA issues addressed? How? Internet and/or connection problems: Facet 11: Publicize – “Base: Create and post publicly a description of the purpose and practices of the service.”
Instructions/tips for use (Print): Hours of operation: Visibility of hours (Where are the hours posted?): Facet 8: Private – “Base: Deny public access to user names, e-mail or mailing addresses, and questions unless there is written notice prior to the information being made available…”
Confidentiality/privacy statements available? (Print):
Where is it located? Facet 3: Clear Response Policy – “Base: Create and adhere to a clear response policy.”
Is the policy posted in a visible location? Where? Facet 10: Provides access to related information – “Base: Include basic resources on the Web site to supplement the question-answering component.” (Ex. FAQs or research guides)
Supplementary resources:
77
Site Number
Virtual Reference Interaction Form
The following script is an outline of the virtual reference conversation. While additional exchanges may be necessary during the reference interview, the exchanges will be kept to a minimum in order to maximize consistency between sites. In the spaces provided, capture the library’s response.
Introduction Line: “Hello, I am in the middle of writing a paper and have hit a dead end on
some information.” <<Wait for intro/welcome type reply to confirm that someone is indeed on
virtual reference>>
Pose Question: “When was the National Institute of Diabetes and
Diseases (NIDDK) founded? I know it has gone through many nam
<<Wait for the response>>
Follow-up Question: “What branch of the U.S. Public Health Serv
the branch founded?” <<Wait for the response>>
Closing: “Thank you for your assistance. Is there a way for me to
so that I have the web addresses? Can I get your email address for
topic?” <<Wait for the response>>
“Thanks again, goodbye.”
Sent:
Received:
Digestive and Kidney
e changes. “
Sent:
Received:
ice is it in, and what year was
Sent:
Received:
get a copy of this conversation
further questions on this
Sent:
Received:
Start Time:
End Time:
78
Site Number
Virtual Reference Evaluation Form
Facet 2: Prompt Turnaround – “Base: Questions should be addressed as quickly as possible.”
How long did the responses take? First log-in response: Subsequent responses: Correctness of answers: Ease of understanding: Appropriate responses for patron level of interest and expertise: Facet 5: Instructive – “Base: Offer answers or pointers to information in responses to users. When unable to provide an answer, provide the user with appropriate notification.”
Answers based on authoritative sources: Avenues for further investigation and follow-up provided: Instruction and resources provided while providing the answer: Facet 4: Interactive – “Base: Encourage sharing of important user information…and question subject-area…” (i.e. effective reference interview)
Is the reference interview effective?
Does the interaction have a personal touch?
Recommended