Making Library Assessment Work

Progress Report of an ARL ProjectSteve Hiller

University of Washington

Jim Self University of Virginia

Martha KyrillidouAssociation of Research Libraries

ARL MLAW Participants MeetingSan Antonio, Texas

20 January 2006

Making Library Assessment Work: Practical Approaches for

Developing and Sustaining Effective Assessment

• Association of Research Libraries Project– Under the aegis of Statistics and Measurement Program

• Funded by participating libraries• Site visits by Jim and Steve

– Pre-visit survey– Presentation– Interviews and meetings – Written report for each library

• Phase I: 7 libraries in Winter/Spring 2005 – Preliminary report at Northumbria Conference 2005

• Phase II: 18 libraries Autumn 2005-Autumn 2006 • Final report to ARL in late 2006

Why “Making Library Assessment Work”?Using Data Effectively in Libraries

• Library leadership• Organizational culture• Library priorities• Sufficiency of resources• Data infrastructure• Assessment skills and expertise • Sustainability • Analyzing and presenting results• Using results to improve libraries

MLAW Process

• Idea discussed with Martha, June 2004• Final proposal written with Martha, August 2004• Invitation to participate sent by ARL in Sept. 2005• 17 libraries express interest; 7 chosen for Phase I,

remainder for Phase II• Pilot site visit November 2005• Second call for participants in June 2005; 8 more

libraries express interest• Phase II revised to include longer site visit, follow-up

project, and community building

The Geographic Distribution of Participants(Jim and Steve Earn Frequent Assessment Miles)

Phase IIParticipant

Phase I Participant

Other ARL Libraries

Steve’s HomeUniversity of Washington

Jim’s HomeUniversity of Virginia

Canada

USA

Distribution Of Participants by ARL Index Ranking (113 Academic Libraries)

24 Participating Libraries: Mean 51.5 Median 49.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

0 10 20 30 40 50 60 70 80 90 100 110

Index Rank

Ind

ex

Sc

ore Non-participants Participants UVA UW

Phase I (At time of visit) Assessment Organization Structure

Assessment Position

Assessment Committee

Contact Person Admin. Area

Other Related Groups

Arizona Administration Distributed within team structure

Arizona State

Services

Connecticut Collection Services User Team, Network Svcs,

Design & Usability

Illinois Services Services Advisory

User Ed, Coll Development

New York U Established 2005

Public Services

Notre Dame Public Services Usability Group

Libraries Task Force

Oregon Administration

Phase II: Autumn 2005 (At time of visit)

Assessment Organization StructureAssessment

PositionAssessment Committee

AdministrativeArea

Other OngoingGroups

Emory 2005 2005 Administration

Louisville 1997 1997 Administration

Kansas 2004 2003 Administration

Massachusetts 2004 2004 Administration

Wayne 2005 2005 Administration

Pre-Visit Survey

• Summary of recent assessment activity • Inventory of statistics kept • Important assessment motivators• Organizational structure for assessment • What has worked well• Problems or sticking points • Specific areas to address• Expectations for this effort• Follow-up project (Phase II)

Phase I:Library-Identified Assessment Needs

Data Collection

Data Analysis

DataUse

Skills & Abilities

Perform. Measures

Data Warehouse

Sustain Assessment Culture

Library 1

Library 2

Library 3

Library 4

Library 5

Library 6

Library 7

Phase II: Autumn 2005Library-Identified Assessment Needs

Data Collection

Data Analysis

DataUse

Skills & Abilities

Perform. Measures

Learning Outcomes

Sustain Assessment Culture

Library 1

Library 2

Library 3

Library 4

Library 5

Sample Site Visit Schedule

• Meet with University Librarian/Contact person• Presentation on effective assessment

– 90 minutes to 2 hours with Q&A• Concepts and best practices• Examples from UVA and UW Libraries

• Group Meetings– with management/administrative group

– assessment-related group (if formed)

– different departments

– functional areas/groups (e.g. info literacy)

• Wrap-up session

Presentation Slide That Fostered Most Discussion

University of Virginia Balanced Scorecard

Metric U.3.A – Circulation of New Monographs

• Target1: 60% of newly cataloged cataloged monographs should circulate within two years

.• Target2: 50% of new monographs should circulate

within two years.

Our Perception of Visits

• Diverse organizational cultures offers opportunities and challenges for successful assessment– Every library is unique

• Reception overwhelmingly positive– Spirited and engaged discussions

• More assessment work going on than being reported– Internally and externally

• Important assessment catalysts include:– accreditation, facilities renovation, student learning, data driven

administrations, LibQUAL+™ results, “should be doing this” • Increase in interest reflected in creation of new assessment

positions and groups

Sample Report Format

• Introduction• Current Assessment Environment and Activities• Identified Issues and Concerns• Suggestions and Options for Moving Forward • Conclusion

Typical Recommendations

• Coordination of assessment– Involve library staff, better communication

• Prioritize assessment activities• Move from projects to sustainable assessment• Share assessment results• Allocate sufficient resources • Review internal statistics • Incorporate data into library management

– Management information systems important

• More knowledge of on-campus activities – Data warehousing

Feedback on Phase I ProcessChanges for Phase II

• One day is too short– Site visit increased to 1.5 days

• Resource materials would be helpful– Web site strengthened

– Assessment “reader” provided

• Follow-up activity would maintain momentum– Consult on follow-up activity (plan, specific effort)

• Establish community of assessment practitioners– Meeting at ALA

– Assessment Conference, September 2006

– ARL Assess list serv

What’s Next? Building the Library Assessment Community

• Meetings– ALA New Orleans, June 2006– Assessment Conference, Charlottesville VA

September 25-27, 2006

• Web Site– http://www.arl.org/stats/Hiller_Self.html

• Final Report– Late 2006