Effective, Sustainable and Practical Library Assessment

Steve Hiller Director, Assessment & Planning, University of Washington Libraries

ARL Visiting Program Officer

Jim SelfDirector, Management Information Services, University of Virginia Library

ARL Visiting Program Officer

University of South Carolina University Libraries December 3, 2007

Library AssessmentMore than Numbers

Library assessment is a structured process:

• To learn about our communities

• To respond to the needs of our users

• To improve our programs and services

• To support the goals of the communities

Why Assess?

• Accountability and justification• Improvement of services • Comparisons with others• Identification of changing patterns• Marketing and promotion • Opportunity to tell our own story• Using data, not assumptions, to make decisions

– Assumicide!

The Challenge for LibrariesThe Challenge for Libraries

• Traditional statistics are no longer sufficient– Emphasize inputs – how big and how many– Do not tell the library’s story– May not align with organizational goals and plans– Do not measure service quality

• Need measurements from the user’s perspective

• Need the organizational culture and the skills to answer a basic question:

What difference do we make to our communities?

ARL Sponsored Assessment

• Tools– ARL Statistics

– LibQUAL+®– MINES for Libraries

• Building a Community of Practice– Library Assessment Conferences

– Service Quality Evaluation Academy

– Library Assessment blog

– Workshops

Individual Library Consultation (Jim and Steve)– Making Library Assessment Work (24 libraries in 2005-06)

– Effective, Sustainable, Practical Library Assessment (6 in 2007)

What We Found in Our Visits

• Strong interest in using assessment to improve customer service and demonstrate value

• Uncertainty on how to establish and sustain assessment• Lack of assessment knowledge among staff• More data collection than data utilization• Effectiveness not dependent on library size or budget• Each library has a unique culture and mission

Effective Assessment

• Focuses on the customer • Is aligned with library and university goals• Assesses what is important• Is outcomes oriented• Develops criteria for success• Uses appropriate and multiple assessment methods• Uses corroboration from other sources• Provides results that can be used

Sustainable Assessment Needs . . • Organizational leadership• Sufficient resources• Supportive organizational culture• Identifiable organizational responsibility • Connection to strategic planning and priorities• Iterative process of data collection, analysis, and use • Involvement of customers, staff and stakeholders

Practical Assessment • Keep it simple and focused – “less is more”• Know when enough is enough• Use assessment that adds value for customers• Present results that are understandable• Organize to act on results

Customer-Centered Library and the Culture of Assessment

Customer-Centered Library • All services and activities are

viewed through the eyes of the customers

• Customers determine quality

• Library services and resources add value to the customer

Culture of Assessment • Organizational environment in

which decisions are based on facts, research and analysis,

• Services are planned and delivered to maximize positive customer outcomes

It’s not about us! It’s about the customer.

Understanding our Customers: What We Need to Know to Support Our Communities

• What are their teaching, learning, and research interests? • How do they work? What’s important to them?• How do they find information needed for their work?• How do they currently use library/information services? • How would they prefer to do so? • How do they differ from each other in library use/needs?

How does the library add value to their work?

How does the library contribute to their success?

If It Was Only This Easy!

Good Assessment Starts Before You Begin . . . Some Questions to Ask

• Define the question– What do you need to know and why

• How will you use the information/results• Where/how will you get the information

– Methods used

– Existing data

– New data (where or who will you get it from)

• How will you analyze the information• Who will act upon the findings

University of Washington(Site of the 2008 Library Assessment Conference!)

• Located in beautiful Seattle metro population 3.2 million

• Comprehensive public research university – 27,000 undergraduate students– 12,000 graduate and professional

students (80 doctoral programs)– 4,000 research and teaching

faculty

• $800 million annually in federal research funds (2nd in U.S.)

• Large research library system– $40 million annual budget– 150 librarians on 3 campuses

University of Washington Libraries Assessment Methods Used

• Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007

• In-library use surveys every 3 years beginning 1993• Focus groups/Interviews (annually since 1998)• Observation (guided and non-obtrusive)• Usability• Usage statistics/data mining

Information about assessment program available at:

http://www.lib.washington.edu/assessment/

Case Study: UW Libraries Review of Support of Bioscience Programs

Reasons for review• Better understand how bioscientists work• Understand significance and value of bioscience and

research enterprise to University• Gauge extent and impact of interdisciplinary research • Understand implications of changes in library use patterns• Review viability of Libraries organizational

structure/footprint • Strengthen library connection in support of bioscience

programs and the research enterprise

The Importance of the Research Enterprise University of Washington Operating Revenues

$2.4 Billion in 2005-06

Tuition 16%

Other 4%

Gifts9%

State Appropriation

15%

Investment Income 13%

Research Grants 43%

Research Grants $1 BillionHealth and Human Services

$510 millionNational Science Foundation

$95 millionOther federal agencies

$190 millionIndustry/Foundations

$100 millionOther non-federal

$110 million

More Than Surveys and StatisticsThe Qualitative Often Provides the Key

• Increased use and importance of such qualitative methods as, comments, interviews, focus groups, usability, observation

• Statistical data often can’t tell us – Who, how, why– Value, impact, outcomes

• Qualitative provides information directly from users– Their language– Their issues– Their work

• Qualitative provides context and understanding

Biosciences Review Process (2006)

• Define scope (e.g. what is “bioscience”?)• Identify and mine existing data sources

– Extensive library assessment data• Including usage information

– Institutional and external data• Including bibliometric information

• Acquire new information through a customer-centered qualitative approach – Environmental scan – Interviews (12 faculty)– Focus groups (6 total – 3 faculty, 2 grad, 1 undergrad) – Peer library surveys

• NO NEW USER SURVEYS

Faculty Interview Key Findings

• First stop: Google or Pub Med Central; also WOS• Those with grant support tend to buy books from Amazon• The transaction costs from discovery to delivery is too high• Need to integrate fragmented library systems and processes• Graduate students are self-sufficient in finding information• Faculty who teach undergraduates use libraries differently• Had difficulty coming up with “new services” unprompted

Focus Group Themes• Content is primary link to the library

– Identify library with ejournals; want more titles & backfiles

• Print is dead, really dead– If not online want it delivered online

• Provide library-related services and resources in our space not yours– Discovery begins outside of library space with Google and

Pub Med; lesser use of library bibliographic data bases

– Faculty/many grads go to physical library as last resort; too many physical libraries

• Lack understanding of many library services and resources

Biosciences Task Force Recommendations

• Integrate search/discovery tools into users workflow• Expand/improve information/service delivery options• Make physical libraries more inviting/easier to use

– Consolidate libraries, collections and service points

– Reduce print holdings; focus on services and work space

• Use an integrated approach to collection allocations• Get librarians to work outside library space• Lead/partner in scholarly communications and E-science • Provide more targeted communication and marketing

In God We Trust: All Others Must Bring Data

Did themes raised in the interviews/focus groups reflect the bioscience population? The campus community? The 2007 Triennial Survey as corroborating source

Related Questions

Mode of access (in-person, remote)

Resource type importance

Sources consulted for research

Primary reasons for using Libraries Web sites

Libraries contribution to work and academic success

Useful library services (new and/or expanded)

UW Triennial Library Survey Number of Respondents and Response Rate 1992-2007

2007 2004 2001 1998 1995 1992

Faculty 1455

36%

1560

40%

1345

36%

1503

40%

1359

31%

1108

28%

Grad

Student

580

33%

627

40%

597

40%

457

46%

409

41%

560

56%

Undergrad 467

20%

502

25%

497

25%

787

39%

463

23%

407

41%

Mode of Library Use by Group 2007 (weekly or more often)

Visit Only 2%

Visit Only 1%

Visit Only 27%

Remote & Visit 19%

Remote & Visit 39%

Remote & Visit45%

Remote Only72%

Remote Only47%

Remote Only14%

Non- Weekly 8%

Non- Weekly6%

Non- Weekly 20%

Faculty

Grad

Undergrad

Library as Place Change In Frequency of In-Person Visits 1998-2007 (weekly+)

Faculty

Faculty

Grad

Grad

Undergrad

Undergrad

20%

30%

40%

50%

60%

70%

80%

1998 2001 2004 2007

Change in Off-Campus Remote Use 1998-2007 (Percentage using library services/collections at least 2x week)

Faculty

Undergrad

Grad

0%

10%

20%

30%

40%

50%

60%

70%

1998 2001 2004 2007

0%

10%

20%

30%

40%

50%

60%

70%

Graduate Student Mode of Accessby Academic Area(% Using at least 2x week)

0%

10%

20%

30%

40%

50%

60%

70%

80%

Health Sciences BioSciences Phys Sci-Engineering Hum-Soc Sciences

Visit In Person Connect from Campus Connect Off-Campus

Where Do they Go? Sources Consulted for Information on Research Topics (Scale of 1 “Not at All” to 5 “Usually”)

Faculty

Grad

Undergrad

2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5

BibliographicDatabases

Open Internet RefSource

Open Internet Search

“If it’s not on the Internet, it doesn’t exist.” My students at all levels behave this way. They also all rely on Wikipedia almost exclusively for basic information. Associate Professor, English

Faculty: Resource Type Importanceby Academic Area (Scale of 1 “not important” to 5 “very important”)

Journals>1985

2.5

3

3.5

4

4.5

5

Health Sciences BioSci Phy Sci-Engin Hum-Soc Science

Books Journals<1985 Bib Databases Journals>1985

Reasons for Faculty Use of Libraries Web Sites by Academic Area (Use at least 2x per week)

15%

25%

35%

45%

55%

65%

75%

Health Sci BioSci Phy Sci-Engin Hum-Soc Sci

Library Catalog Bib Database Online journal articles

Usefulness of New/Expanded Services for Faculty & Grads

0% 10% 20% 30% 40% 50% 60% 70% 80%

Manage your info anddata

Integrate services intocampus Web sites

Office Delivery ofBooks

Digitize specializedcollections

Scan on Demand

Grad Faculty

Libraries Contribution to: (Scale of 1 “Minor” to 5 “Major”)

3 3.25 3.5 3.75 4 4.25 4.5 4.75

Academic Success

Efficient Use of Time

Finding info in new orrelated areas

Keeping current in yourfield

Being a more productiveresearcher

Grad Faculty

Survey Follow-Up Actions

• Probe deeper on specific library contributions to research and student academic success using qualitative methods– Interviews/focus groups beginning Winter 2008– Review scope and effectiveness of information literacy programs

• Develop plan to deliver “print” content to faculty & grad students in their format of choice and in their space– Pilot test “scan on demand” begins January 2008

• Strengthen our subject librarian liaison efforts to better understand and support research in their areas– Develop standardized toolkit for assessing library connection to

research enterprise. Revisit scholarly communications policy

• Integrate library services & resources into user workflows

How UW Libraries Has Used Assessment

• Extend hours in Undergraduate Library (24/5.5)• Create more diversified student learning spaces• Eliminate print copies of journals • Enhance usability of discovery tools and website• Provide standardized service training for all staff• Stop activities that do not add value • Change/reallocate budgets and staffing• Inform the strategic planning process• Support budget requests to University

How Are We Doing?Overall Satisfaction by Group 1995-2007

Faculty 4.25

4.44

4.33 4.33

Faculty 4.56

Undergrad 3.97 3.99

4.22

4.32UW Seattle UG

4.36

4.34

4.26

4.11Grad 4.18

Grad 4.36

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

1995 1998 2001 2004 2007

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

You guys and gals rock!!!!!! We need to invest in our library system to keep it the best system in America. The tops! My reputation is in large part due to you. Professor, Forest Resources

UVA MIS Consult

Collecting the Data at U.Va.

• Customer Surveys

• Staff Surveys

• Mining Existing Records

• Comparisons with peers

• Qualitative techniques

Corroboration

• Data are more credible if they are supported by other information

• John Le Carre’s two proofs

UVa Customer Surveys

• Faculty – 1993, 1996, 2000, 2004

– Separate analysis for each academic unit

– Response rates 59% to 70%

• Students – 1994, 1998, 2001, 2005

– Separate analysis for grads and undergrads

– Undergrad response rates 43% to 50%

– Grad response rates 54% to 63%

LibQUAL+™ in 2006

Analyzing U.Va. Survey Results

• Two Scores for Resources, Services, Facilities– Satisfaction = Mean Rating (1 to 5)– Visibility = Percentage Answering the Question

• Permits comparison over time and among groups

• Identifies areas that need more attention

Reference Activity and Visibility in Student Surveys

1,756

6,008

34%Visibility

39% Visibility

75% Visibililty

64% Visibility

1,000

7,000

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Re

fere

nc

e Q

ue

sti

on

s

Re

co

rde

d p

er

We

ek

in

An

nu

al S

am

ple

10%

Re

fere

nc

e V

isib

ilit

y

am

on

g U

nd

erg

rad

ua

te

Making the most of LibQUAL

• Scan the results by user category

• Use thermometer charts

• Identify high and low desire areas

• Identify the ‘red zones’

• Examine the comments

• Compare satisfaction scores with peers

• Follow up on problem areas

LibQUAL+ 2006University of Virginia Faculty

5.00

6.00

7.00

8.00

9.00

AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

Red Square = Perceived Service Performance

Top of Blue Bar = Desired Level of Service

Bottom of Blue Bar = Minimum Level of Service

LibQUAL+ 2006University of Virginia Library

Areas Needing Attention

6

7

8

9

IC-1 Remote Access Grads IC-2 Website Faculty IC-8 Journals Faculty IC-8 Journals Grads

Red Square = Perceived Service Performance

Top of Blue Bar = Desired Level of Service

Bottom of Blue Bar = Minimum Level of Service

Grads Remote Access

FacultyLibrary Website

Faculty Journals

Grads Journals

LibQUAL+ 2006University of Virginia

Faculty Ratings of Journal Collections

5.00

6.00

7.00

8.00

9.00

UVA Faculty Overall Architecture Faculty Education Faculty Engineering Faculty Humanities Faculty Science/Math Faculty Social Science Faculty

Red Square = Perceived Service Performance

Top of Blue Bar = Desired Level of Service

Bottom of Blue Bar = Minimum Level of Service

LibQUAL Follow Up on Journals

• Examining the comments

• Drilling into data

• Talking to faculty and grad students

• Corroborating with other data

• Comparing with other libraries

2006 LibQUAL+™ ResultsUVa and ARL Overall Satisfaction

Undergrad

Overall

Grad Overall Faculty

Overall

UVa 7.52 7.48 7.87

ARL

Range

6.61 to

7.63

6.51 to

7.63

5.87 to

7.87

ARL

Mean

7.18 7.16 7.24

The Balanced ScorecardManaging and Assessing Data

• The Balanced Scorecard is a layered and categorized instrument that– Identifies the important statistics– Ensures a proper balance– Organizes multiple statistics into an intelligible

framework

Metrics

• Specific targets indicating full success, partial success, and failure

• At the end of the year we know if we have met our target for each metric

• The metric may be a complex measure encompassing several elements

What Do We Measure?

• Customer survey ratings

• Staff survey ratings

• Timeliness and cost of service

• Usability testing of web resources

• Success in fundraising

• Comparisons with peers

Metric U.1.A: Overall Rating in Student and

Faculty Surveys.• Target1: A score of at least 4.00 (out of 5.00)

from each of the major constituencies.

• Target2: A score of at least 3.90 from each of the major constituencies.

• FY05 Result: Target1 – Undergraduates 4.08

– Graduate Students 4.13

Metric U.4.B: Turnaround time for user requests

• Target1: 75% of user requests for new books should be filled within 7 days.

• Target2: 50% of user requests for new books should be filled within 7 days.

• Result FY06: Target1.– 79% filled within 7 days.

Metric U.3.A: Circulation of New Monographs

• Target1: 60% of newly cataloged monographs should circulate within two years.

• Target2: 50% of new monographs should circulate within two years.

• Result FY06: Target1.– 61% circulated.

Using Data for Results at UVa

• Additional resources for the science libraries (1994+)

• Redefinition of collection development (1996)

• Initiative to improve shelving (1999)

• Undergraduate library open 24 hours (2000)

• Additional resources for the Fine Arts Library (2000)

• Support for transition from print to e-journals (2004)

• New and Improved Study Space (2005-06)

in conclusion Assessment is not…

• Free and easy• A one-time effort• A complete diagnosis• A roadmap to the future

Assessment is…

• A way to improve • An opportunity to know our customers• A chance to tell our own story• A positive experience

Moving Forward

• Keep expectations reasonable and achievable• Strive for accuracy and honesty—not perfection• Assess what is important• Use the data to improve • Keep everyone involved and informed• Focus on the customer

For more information…

• Steve Hillerhiller@u.washington.eduwww.lib.washington.edu/assessment/

• Jim Self– self@virginia.edu– www.lib.virginia.edu/mis– www.lib.virginia.edu/bsc

• ARL Assessment Servicewww.arl.org/stats/initiatives/esp/