Global Developments in Assessment: LibQUAL+ ® and Beyond… 6th International JISC/CNI Conference York, U.K. July 6-7, 2006 Duane Webster Executive Director

Global Developments in Assessment: LibQUAL+® and Beyond…

6th International JISC/CNI ConferenceYork, U.K.

July 6-7, 2006

Duane WebsterExecutive Director

Association of Research Libraries

Selena LockResearch and Development

Cranfield University

ARL www.arl.org

Duane Webster, Executive Director

What are some of the current developments with library assessments efforts?

ARL StatsQUAL™

E-Metrics

LibQUAL+®

DigiQUAL™

MINES for Libraries™

Where are the most critical needs and opportunities?

What are the lessons learned?

Library Assessment in an Electronic Era

www.arl.org


Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication.

Members: 123 major research libraries in North America.

Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources.

Users: Three million students and faculty served.

Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 31 percent is invested in access to electronic resources.

www.arl.org


Thinking Strategically About Library Futures

• What is the central work of the library and how can we do more, differently, and at less cost?

• What important set of services does the library provide that others can’t? What new roles are needed?

• What advantages does the research library possess?

• What will be the most needed by our community of users in the next decade? How is user behavior changing?

• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?

• What are the essential factors responsible for the success of the library?

www.arl.org


Defining Success in a Digital Environment

• Crafting new measures of success.

• Moving from measuring inputs to outputs.

• Understanding impact of library roles and services.

• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.

• Reallocating and managing capabilities to focus on new definitions of success.

www.arl.org


ARL Statistics and Measurement

www.arl.org/stats/


.

StatsQUAL™

Duane Webster, Executive Director www.statsqual.org


StatsHome

LibQUAL+DigiQUAL(/digiqual)

MINES(/mines)

SAILS(/sails)

E-Metrics(/emetrics)

ARL Statistics(/arlstats)

Interactive Statistics

(/interactive)?

Login

UserProfile

InstitutionProfile

www.libqual.org

SurveyManagement

StatsQUAL™

www.statsqual.org


Updating the Traditional ARL Statistics

• E-Metrics = ARL Supplementary Statistics– On going efforts to update and refine core data.

– Exploring feasibility of collecting e-metrics.

• ARL Task Force on New Ways of Measuring Collections :– Growing concern with utility of membership index.

– Study ARL statistics to determine relevance.

– Develop Profile of Emerging Research Libraries.

www.statsqual.org


E-Metrics: Recommended Statistics

Patron Accessible Electronic Resources (R1-3)

Use of Networked Resources and Services (U1-5)

Expenditures for Networked Resources and Related Infrastructure (C1-3)

Library Digitization Activities (D1-3)

www.arl.org/stats/newmeas/emetrics/index.html



Patron Accessible Electronic Resources (R1-3)

• R1 – Number of electronic full-text journals

• R2 – Number of electronic reference sources

• R3 – Number of electronic books




Use of Networked Resources and Services (U1-5)

• U1 – Number of electronic reference transactions

• U2 – Number of logins (sessions) to electronic databases

• U3 – Number of queries (searches) in electronic databases

• U4 – Items requested in electronic databases

• U5 – Virtual visits to library’s website and catalog



Expenditures for Networked Resources and Related Infrastructure (C1-3)

• C1 Cost of electronic full-text journals

• C2 Cost of electronic reference sources

• C3 Cost of electronic books

• C4 Library expenditures for bibliographic utilities, networks and consortia

• C5 External expenditures for bibliographic utilities, networks and consortia




Library Digitization Activities

Library Digitization Activities (D1-3)

• D1 – Size of library digital collection

• D2 – Use of library digital collection

• D3 – Cost of digital collection construction and management(Collecting these data requires staff

familiar with the digital environment.)



A LibQUAL+® Update

• The LibQUAL+® premise, dimensions, and methodology

• LibQUAL+® results

• LibQUAL+ in action

www.libqual.org

http://www.libqual.org

LibQUAL+™

Presented by:

Selena LockSelena LockCranfield UniversityCranfield University

6th InternationalJISC/CNI Conference York, U.K. July 6-7, 2006

A A LibQUAL+LibQUAL+®® Update Update


A LibQUAL+® Update

The LibQUAL+® premise, dimensions, and methodology

LibQUAL+® resultsLibQUAL+® in action


LibQUAL+™

The LibQUAL+® premise, dimensions, and methodology


The need for LibQUAL+®

Underlying need to demonstrate our worthThe reallocation of resources from

traditional services and functionsRapid shifts in information-seeking

behavior• Need to keep abreast of customer demands

Increasing user demands• 37% of UK 16 – 18 year olds expect better

libraries in return for their top-up fees


PERCEPTIONS SERVICE

“….only customers judge quality;

all other judgments are essentially

irrelevant”

Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

The LibQUAL+® Premise


Multiple Methodsof Listening to Customers

Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture*A SERVQUAL-type instrument is most suitable for these methods

Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000).

Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.


LibQUAL+® Development

An ARL/Texas A&M University joint developmental effort based on SERVQUAL

LibQUAL+® initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE)

Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology

Survey conducted at over 700 libraries resulting in a data base of over a million user responses

13 LibrariesEnglish LibQUAL+® Version

4000 Respondents

QUAL

QUAN

QUAL

QUAL

QUAN

QUAL

PURPOSE DATA ANALYSIS PRODUCT/RESULTDescribe library environment;build theory of library service quality from user perspective

Test LibQUAL+® instrument

Refine theoryof service quality

Refine LibQUAL+® instrument

Test LibQUAL+® instrument

Refine theory

Unstructured interviewsat 8 ARL institutions

Web-delivered survey

Unstructured interviews at Health Sciences and the Smithsonian libraries

E-mail to surveyadministrators

Web-delivered survey

Focus groups

Content analysis:(cards & Atlas TI)

Reliability/validityanalyses: CronbachsAlpha, factor analysis,SEM, descriptive statistics

Content analysis

Content analysis

Reliability/validity analyses including Cronbachs Alpha,factor analysis, SEM, descriptive statistics

Content analysis

VignetteRe-tooling

Iterative

Emergent2000

2005700 Libraries English, Dutch, Swedish,

German LibQUAL+® Versions160,000 anticipated respondents

LibQUAL+LibQUAL+® Project® Project

Case studies1

Valid LibQUAL+® protocol

Scalable process

Enhanced understanding of user-centered views of service quality in the library environment2

Cultural perspective3

Refined survey delivery process and theory of service quality4

Refined LibQUAL+® instrument5

Local contextual understanding of LibQUAL+® survey responses6


76 Interviews Conducted

York UniversityUniversity of ArizonaArizona StateUniversity of ConnecticutUniversity of HoustonUniversity of Kansas

University of MinnesotaUniversity of

PennsylvaniaUniversity of WashingtonSmithsonianNorthwestern Medical

LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred


Dimensions ofLibrary Service Quality

Information Control

LibraryServiceQuality

Model 3

Self-Reliance

Equipment

Timeliness

Ease of Navigation

Convenience

Scope of Content

Affect of Service

Library as Place

Reliability

Assurance

Responsiveness

Empathy

Refuge

Symbol

Utilitarian Space


LibQUAL+® Participants

12

43

164

316

207

255

0

50

100

150

200

250

300

350

2000 2001 2002 2003 2004 2005


Rapid Growth in Other Areas

Languages• American English• British English• French• Dutch• Swedish

In development• Chinese• Greek• Spanish• German

Consortia• Each may create 5 local

questions to add to their survey

Types of Institutions• Academic Health Sciences• Academic Law• Academic Military• College or University• Community College• European Business• Hospital• Public• State

Countries• U.S., U.K., Canada, the

Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

Participating Libraries

World LibQUAL+® Survey 2005


LibQUAL+® Languages

American English

Dutch EnglishFrench Canadian DutchSwedish

Swedish(British English)

Afrikaans

DanishFinnishGerman Norwegian

British English

Continental French

Over 700 institutions 1,000,000 respondents


Survey Instrument – “22 items…


…and a Box”

Why the Box is so Important:• About 40% of participants provide open-

ended comments, and these are linked to demographics and quantitative data

• Users elaborate the details of their concerns• Users feel the need to be constructive in their

criticisms, and offer specific suggestions for action


LibQUAL+™

LibQUAL+® results


Understanding LibQUAL+®

Results

For the 22 items LibQUAL+ asks users’ to rate their:• Minimum service level• Desired service level• Perceived service performance

This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings

Perception ratings ideally fall within the Zone of Tolerance


Key to Bar Charts Key to Radar Charts


LibQUAL+® 2005 Summary Collegesor Universities (American English)


LibQUAL+® 2005 Summary Collegesor Universities (British English)


Longitudinal Analysis for Collegesor Universities (American English)

5.50

6.00

6.50

7.00

7.50

8.00

8.50

2003 2004 2005

Minimum Mean

Desired Mean

Perceived Mean


Longitudinal Analysis for Collegesor Universities (British English)

5.50

6.00

6.50

7.00

7.50

8.00

8.50

2003 2004 2005

Minimum Mean

Desired Mean

Perceived Mean


Comparisons by user groupCollege or University (British English)

5.50

6.00

6.50

7.00

7.50

8.00

8.50

2003 2004 2005

Minimum Mean

Desired Mean

Perceived Mean


General findings

Highly desired• Making electronic resources accessible from my

home or office• Print and/or electronic journals I require for my work• A haven for study, learning or research

Lowest • Library staff who instil confidence in users• Giving users individual attention• Space for group learning and group study


LibQUAL+™

LibQUAL+® in Action


UK University Case Study – Using LibQUAL+® Results

Strategic Service Developments• Data to support service development• Ability to identify where not meeting expectations• Measure if change has met need

Budget Discussions• Data to support bid for increased funding• Data to support case for change in emphasis (towards

e-provision)Marketing Position

• Status of the library within the University• Importance of national & international benchmarking


LibQUAL+® Outcomes

Funding received for:New Web Services AdministratorIncreased opening hours

• Now providing 222,578 seat hours per week

Library refurbishment programme reinstated at costs in excess of £8 million


Library Refurbishment

From: To:


Why use LibQUAL+®?Feedback from LibQUAL+® Users

“Why did you choose to use LibQUAL+®?”LibQUAL+® was recommended to us as offering

a well designed, thoroughly Library-focused set of survey tools

Opportunity to benchmarkCost-effectivenessAutomated processing & fast delivery of resultsRespectability and comparability (with others

and historically)


The benefits of LibQUAL+®

LibQUAL+® has enabled us to find out what a broad range of our users thought of the services we offer; what level of service-delivery quality we had achieved in their eyes, and to get a clear picture of what they actually wanted the Library to deliver (as opposed to what we thought they wanted).

UK HE Institution, 2006


In Closing LibQUAL+®

Focuses on success from the users’ point of view (outcomes)

Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense

Requires limited local survey expertise and resources Analysis available at local, national and inter-institutional

levels Offers opportunities for highlighting and improving your

status within the institution Can help in securing funding for the Library


LibQUAL+® Resources

LibQUAL+® Website:http://www.libqual.org

Publications: http://www.libqual.org/publications

Events and Training: http://www.libqual.org/events

LibQUAL+® Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib

LibQUAL+® Procedures Manual: http://www.libqual.org/Information/Manual/index.cfm

http://www.libqual.org/

http://www.libqual.org/publications

http://www.libqual.org/events

http://www.coe.tamu.edu/~bthompson/servqbib

http://www.coe.tamu.edu/~bthompson/servqbib

http://www.libqual.org/Information/Manual/index.cfm

http://www.libqual.org/Information/Manual/index.cfm


Acknowledgements

Fred Heath, Bruce Thompson, Colleen Cook, Martha Kyrillidou and the rest of the LibQUAL+® team

Stephen Town, Cranfield UniversityHelen Durndell and Jacqui Dowd, Glasgow

UniversityAll SCONUL LibQUAL+® Participants and the

SCONUL Working Group on Performance Improvement


LibQUAL+® Contact Information

Martha Kyrillidou• Director, ARL Statistics and Measurement Program

• [email protected]

Selena Lock• Research and Development, Cranfield University

• [email protected]

mailto:[email protected]

mailto:[email protected]

Developing the DigiQUAL™ Protocol forDigital Library Evaluation

www.digiqual.org

DigiQUAL™

• NSF Funding

• Building on the LibQUAL+® experience

• Secures feedback on user’s perceptions of library’s web site

• Five questions on services, functionality, and content

• Goal is to determine utility, reliability, and trustworthiness


Background: ServQUAL LibQUAL+® DigiQUAL™

LibQUAL+® Dimensions of Service Quality:• Affect of Service• Information Control• Library as Place

Developing DigiQUAL™ Survey Items

DigiQUAL™12 themes of service quality:• Accessibility• Navigability• Interoperability• Collection building• Resource Use• Evaluating collections• DL as community for users• DL as community for developers• DL as community for reviewers• Copyright• Role of Federations• DL Sustainability

www.digiqual.org


Pilot Testing Survey Items and Implementation

www.digiqual.org


Building a Survey

• Review and select items

• Issue: aligning items to individual DL needs & users –

vocabulary and content

www.digiqual.org


CustomizeSurvey

Issue: Flexibility vs. Standardization

Building a Survey, cont.

www.digiqual.org


Implementing Survey – Notification Methods

Links on siteNewsletters

Next to resources

Issues: no pop-ups, no individual emailswww.digiqual.org


Implementing Survey - Incentives

Issues: must be easily transferable, requires email address - clear IRB

www.digiqual.org


(mis)Interpreting results from individual digital libraries in the context of other sites

Sites reluctant to sharedata and results

Analysis and Reporting - Issues

www.digiqual.org


Outstanding Issues and Challenges

• Unique DLs: niche market, critical mass, both?

• Balance:– custom vs. generic content results– flexible vs. standard implementation scaling

• Mixed methods– Preserving user privacy– Collecting truly useful data

• Moving target: digital libraries as… it depends.

www.digiqual.org


Assessing the Value of Networked Electronic Services:

Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™

www.arl.org/stats/newmeas/mines.html

The MINES survey


• A research methodology consisting of a web-based survey form and a sampling plan.

• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.

• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.

• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).

What is MINES?



• How extensively do sponsored researchers use the new digital information environment?

• Are researchers more likely to use networked electronic resources from inside or outside the library?

• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

• Are particular network configurations more conducive to studies of digital libraries patron use?

Questions Addressed



• A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.

• Random moment/web-based surveys are employed at each site.

• Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time.

• Libraries with database-to-web gateways or proxy re-writers offer the most comprehensive networking solution for surveying all networked services users during survey periods.

MINES Methodological Considerations



Web Survey Design Guidelines

• Simple text for different browsers – no graphics.– Different browsers render web pages differently.

• Few questions per screen or simply few questions.• Easy to navigate.• Short and plain.• No scrolling.• Clear and encouraging error or warning messages.• Every question answered in a similar way - consistent.

– Radio buttons, drop downs.• ADA compliant.• Introduction page or paragraph.• Easy to read.

– Must see definitions of sponsored research. • Can present questions in response to answers.

Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method.

2nd Ed. New York: John Wiley & Sons. www.arl.org/stats/newmeas/mines.html


Library User Survey



Data was collected at seven main campus libraries and seven academic health science libraries in the U.S., between January, 2003 and January, 2005.

Recent DataCollection Activities

Main University Libraries Academic Medical Libraries

University of Colorado University of Connecticut Health Center

University of Connecticut University of North Carolina

University of North Carolina University of Texas Medical Branch

Oregon State University University of Texas Southwestern

University of Utah University of Utah

University of Virginia University of Virginia

Washington University Washington University



• More than 45,000 networked electronic services uses were surveyed.

• At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.

• MINES is also being conducted by the Ontario Council of University Libraries (OCUL) in 2004-2005 in conjunction with ARL.

Recent DataCollection Activities



Purpose of Use By Location Main Campus Libraries

2003 – 2005

Sponsored Research

Instruction

Other

Other SponsoredActivities

In the Libraryn = 7,618

On-Campus, not in the Libraryn = 6,641

All Usagen = 19,271

•72% of sponsored research usage of electronic resources occurred outside the library; 83% of this took place on campus.

Off-Campusn = 5,012

26%

3%

5%

66%

33%

2%

6%

59%

24%

2%

11%

63%

14%2%

21%

63%



13,930

31,451

Location of Users2003-2005

7,618

11,653

Medical Library Main Library

All Libraries

30%

40%

60%

70%

76%24%

6,312

19,798

In Library Users

Remote Users

76%24%



Demographics by Location of User Main Libraries

On Campus, Not in the Libraryn = 6,391

Inside the Libraryn = 7,064

Off-Campusn = 4,953

Total Usersn = 18,408

20%8%

27%

45%

27%39%

12%

22%

40%3%

33%

24%

38%

62%

In the Library

Outside the Library

Graduate Students

Faculty, Staff, Research Fellows

Undergraduate Students

All Other Users



Electronic Services Sponsored Research Use

Compared to Print Journal and Total Library Use Main Libraries


E-Resources

Print Journals

Total Use


Where are the most critical assessment needs and opportunities?

• Complementing LibQUAL+® with additional measures.

• Developing impact studies on user success, economic value, and community return on investment.

• Moving target: what is a digital library?

• E-Resources: understanding usage.

• Gaining acceptance and use of standard measures for e-resources.

• Building a climate of assessment throughout library.


What are the lessons learned?

• Understanding changes in users approach to information resources.

• Service quality improvement is a key factor.

• Understanding the impact of e-resources on library services - TRL.

• Learning how to compete with Google.

• Upfront investment in design and development.

• Making the assessment service affordable, practical, & effective.

• Assessment needs to be satisfying and fun.


In Closing

• As higher education is challenged on accountability and effectiveness issues so will libraries.

• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.

• Basic questions of role, vision, and impact must be answered by library community.


Selected References

• Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf]

• Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf]

• Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html]

• Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html].

• Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf]

• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf]

• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF]

Selected References

http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf

http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf

http://www.arl.org/newsltr/230/usage.html

http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf

http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf

http://www.libqual.org/documents/admin/ISKO.PDF



Documents

Global Developments in Assessment: LibQUAL+ ® and Beyond… 6th International JISC/CNI Conference York, U.K. July 6-7, 2006 Duane Webster Executive Director