Why assess? What is assessment? What do we mean by actionable data? Martha Kyrillidou Planning for Results: Making Data Work for You The Cato Institute,

Why assess? What is assessment? What do we mean by actionable data?

Martha Kyrillidou

Planning for Results: Making Data Work for You

The Cato Institute, Washington, DC

February 20, 2008

Association of Research Libraries http://www.arl.org/stats

What’s bringing us together …What’s bringing us together …


OverviewOverview

1. Initiate (Why do we assess?)

2. Plan (What do we assess?)

3. Design(How do we assess?)

4. Implement(Design in action)

5. Analyze and Interpret(What do the results mean?)

6. Reflect and Initiate (What do we do with the results?)

“Why assess? What is assessment? What do we mean by actionable data?” - Martha Kyrillidou (1&2)

“Survey Design” – Neal Kaske (2&3)

“Performance Measures” – Rick Garcia (2&3)

“Analysis and Interpretation” - Ray Lyons (5)

LibQUAL+®, MINES™, and Other Tools; Packaging Results to Communicate to Stakeholders Gary Roebuck and Kristina Justh (2,3,4,&5)


Clear and present dangerClear and present danger

• Bangor University considers removing librarians posted by Blake on Thursday January 27, 2005 @07:30AM -753 hits Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all but one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom. In essence, there will remain 4 professional librarians serving a 'research-led' university of 8,000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian.

• Has anything like this happened anywhere that you know of? If so, what have been the effects?

http://eblake.com/

http://www.lisnews.com/print.pl?sid=05/01/27/0719254

http://www.lisnews.com/email.pl?sid=05/01/27/0719254

http://www.lisnews.com/search.pl?topic=26

mailto:[email protected]

Assessment Assessment LibrarianLibrarian


Library of the Future Will Also Need . . .Library of the Future Will Also Need . . .

. . . To have it’s own data collection and management personnel, individuals who constantly collect, analyze and prepare reports on data regarding what services are being used, which portions of the collection are getting the highest usage, what materials are being lent through interlibrary loan, and who patrons are.

Documenting the libraries contributions to quality teaching, student outcomes, research productivity will become critical.


Technical Skills neededTechnical Skills needed

• Quantitative Analysis: SPSS

• Qualitative Analysis: Atlas.ti

• Data collection: Custom Web interface, Survey Monkey, FormSpring

• Data storage: SQL Server, Excel

• Reporting: Crystal Reports, SPSS, SQL Reporting Services, Excel, Word

• Web delivery: Dundas Charts

• Web development: ASP.NET, C#, ColdFusion


Communication Skills neededCommunication Skills needed

• Communicating organizational vision and service ethic

• Leadership in capturing the future and making it happen

• Ability to take initiative and willingness to take risks and learn from mistakes

• Troubleshooting and being an independent thinker

• Ability to work collaboratively and move forward team and organizational goals


. . . chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in.

– President William J. Clinton, on signing the Government Performance and Results Act of 1993


What’s driving the ‘assessment’ agendaWhat’s driving the ‘assessment’ agenda

• “Times are changing”– Exploding growth in use and applications of

technology– Increased customer expectations for services,

including quality and responsiveness

• Shrinking budgets– Justification for spending $$$ on resources,

programs, and services

– Increasing competition for resources– Cost savings and/or reallocation– Fighting to remain relevant


The Internet Goes to CollegeThe Internet Goes to College

Early data from ethnographic interviews– “I use Google because I heard it searches for more

things” (than other sources).– “I believe I can find anything on the Internet. There

hasn’t been anything I haven’t been able to find.” – “Because I’m lazy.”– Books have “so much information that no one can go

through it all.”– I use “the Internet first because it is more convenient.”– I go to the library “because that’s what teachers like.” – “Google has gotten me through college.”

Source: Steve Jones, The Internet Goes to College, ARL Talk


Libraries Remain a Credible Resource in Libraries Remain a Credible Resource in 2121stst Century Century

98% agree with statement98% agree with statement, “My … library , “My … library contains information from credible and contains information from credible and known sources.”known sources.”

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.


OCLC ReportsOCLC Reports

• Sharing, Privacy and Trust in Our Networked World

• College Student’s Perceptions: Libraries and Information Resources

• Perceptions of Libraries and Information Resources

» OCLC: http://www.oclc.org/reports/sharing/default.htm


‘‘Universal’ library brandUniversal’ library brand

• Amazing stability of library user’s desired expectations across– Types of users– Languages– Countries– Some differences across disciplines

• 297,158 LibQUAL+® survey respondents


Towards what endTowards what end

The emerging library user sees the need for a different kind of library – a library that serves as a source of information ready to be accessed any time, from anywhere, using modern technologies. Can we effectively translate the comfort and assurance of human interactions into a virtual reality that is trustworthy, reliable, and ever present? And will learning be quite as effective or deep in this virtual world of never-ending interaction with the collective digital-knowledge base?

• Thompson, Kyrillidou, Cook “Library Users’ Service Desires” Library Quarterly (2008)


Library Assessment and Library Assessment and its Global Dimensionsits Global Dimensions

• Markets and people exposed to economic and social frameworks unheard of before

• Competing internationally

• Library users exposed to global forces

• Libraries facing similar challenges

• Libraries as the Internet

• Libraries as Google

• Libraries as Collaborative Spaces


Defining Success in a Digital EnvironmentDefining Success in a Digital Environment

• Crafting new measures of success.

• Moving from measuring inputs to outputs and outcomes

• Understanding impact of library roles and services.

• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing organizational goals.

• Reallocating and managing capabilities to focus on new definitions of success.


Thinking Strategically About Libraries’ Thinking Strategically About Libraries’ FuturesFutures

• What is the central work of the library and how can we do more, differently, and at less cost?

• What important set of services does the library provide that others can’t? What new roles are needed?

• What advantages does the library possess?• What will be the most needed by our community of users in

the next decade? How is user behavior changing?• What should our libraries aspire to be ten years from now?

What are the implications of technology driven change?• What are the essential factors responsible for the success

of the library?


Library AssessmentLibrary Assessment

Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do

The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities.

Academic libraries do not exist in a vacuum but are

part of a larger institution. Assessment within the institution may take place in individual areas as well as at the broad institutional level.


Assessment is…Assessment is…

“…a structured process to learn about communities and evaluate how well the library supports them.”

- Steve Hiller, Director of Assessment & PlanningUniversity of Washington Libraries


Free speech wall, Free speech wall, Charlottesville, Sept 2006Charlottesville, Sept 2006


Evaluation – foci by Peter BrophyEvaluation – foci by Peter Brophy

Inputs Processes Outputs Outcomes Impacts

Increasing difficulty …………Increasing usefulness …………

We have 2 million records

We have the technology to

transform data into

information-rich products

We have delivered one million

documents

26 students have read this

paper

As a result of

our service, humankind has beaten malaria


IFLA: Measuring QualityIFLA: Measuring Quality

• Resources, infrastructure: What services does the library offer?

• Use: How are the services accepted?

• Efficiency: Are the services offered cost-effectively?

• Potentials and Development: Are there sufficient potentials for future development?

(an adaptation of the Balanced Score Card)


Assessment at ARL

• A gateway to assessment tools: StatsQUAL®:

– ARL Statistics -- E-Metrics

– LibQUAL+® -- Service Quality and Satisfaction

– DigiQUAL®

– MINES for Libraries®

• Library Assessment Conferences – libraryassessment.org• Library Assessment Blog – libraryassessment.info• SPEC Kits• Service Quality Evaluation Academy• ESP Assessment (Making Library Assessment Work)

– Effective, Sustainable, Practical


Qualitative ProfileQualitative ProfileDeveloping New Metrics Developing New Metrics (per Yvonna Lincoln)(per Yvonna Lincoln)

• Uniqueness of collections• Defining the value of consortia• Administrative and budgetary efficiencies• Student outcomes/student learning/graduate success• Contributions to faculty/stakeholders productivity• Social frameworks/intellectual networks• Generating new knowledge• Creating the collective good with reusable assets


How do you describe a Library?How do you describe a Library?

• Breadth and quality of collections and services• Sustained institutional commitment to the library• Distinctive resources in a variety of media• Services to the scholarly community• Preservation of research resources• Contributions of staff to the profession• Effective and innovative use of technology• Engagement of the library in academic planning

Association of Research Libraries ‘Principles of Membership’


Group exerciseGroup exercise

• Share important aspects of your

–Services

–Collections

–Collaborative Relations


Tell the story …Tell the story …

• Facilities: number and type (library buildings, classrooms, studios, computer labs, cafes, offices outside of the library, etc.)

• University centers, services and responsibilities (managed by the library or housed in the library): slide libraries, media centers, teaching and learning centers, education centers, writing centers, distance ed or online course development/instructional design, iTunesU, laptop support, technology help desk, escholarship centers, university press or other publishing entity, university archives, university records management, etc.

• Digital collections: institutional/digital repositories, images, articles, simulations, data, learning objects, etc.

• Grants and development: numbers, types, friends organizations, advisory boards/councils, etc.

• Partnerships: nature and number, on campus and external • Consortial memberships and roles • Outreach: web hits, computer use hours, learning hours/contact/impacts (in

classrooms, Info Lit, etc.), visits, external teaching, volunteering, programs offerings, sponsored lectures, etc.

• Leadership: on campus and external committees, boards, initiatives, projects, etc. Publications/scholarship: numbers, types (articles, books, presentations, poster sessions, performances, etc.)

» Email from ARL Library Director – February 13, 2008


Measuring ImpactMeasuring Impact

• Concerns Based Adoption Model (CBAM)– Stages of concern

• How engaged are subjects in the service /innovation?

• How much interest do they show?

– Levels of use• How much use do they make?

• Avoid dichotomy of user / non-user

– Levels of Impact• Draw together concern and use

• Assess effects on behaviour

» Peter Brophy (South Africa, August 2007)


Measuring ImpactMeasuring Impact

• Surface learning– Skim, memorise, regurgitate for tests

• Little long-term impact

• Deep learning– Involved, questioning, interactive, integration with own

knowledge• Significant long-term impact

• A habit of critical thinking

» Peter Brophy (South Africa, August 2007):


“If you think you're too small to have an impact, try going to bed with a mosquito.”

»Anita Roddick


Actionable Data Actionable Data

• What is the desired outcome of assessment?

– An organization that strives to be customer-centered and acts on information about environmental factors and especially on user input.

» Raynna Bowlby and Martha Kyrillidou


Align with mission, vision and goals of Align with mission, vision and goals of parent organizationparent organization

– Are there prominent themes/directions – research support, instruction, physical facilities; priority user groups; expansion and growth or conservation and reallocation of resources; emphasis on current technologies and methods or more traditional approaches

– Are there strategically favored programs/initiatives and/or individual champions with key agendas?

– Are there stated goals for the organization and the library?


Address users’ top prioritiesAddress users’ top priorities

• Are you undertaking initiatives that users indicate are their highest wants and are you doing enough?

• What budget/resources are being re/allocated to these initiatives?

• Are enough budget/resources being directed to these initiatives?

• Have the departments/staff that most affect these initiatives been informed?

• Do these departments/staff understand the data (and comments/specifics) and have goals and improvement plans in place?

• Are there improvement targets; how is change being measured?


Identify resources that can be reallocatedIdentify resources that can be reallocated

• Is the work that you are performing addressing areas of strong dissatisfaction?

• Are you doing too much of what you’ve always done, rather than the work that results in user wants & needs?

• What budget/resources are being allocated to legacy work that is not a user priority and is it too much?

• Have the departments/staff that are currently doing this work been informed of changing user wants and needs?

• Do these departments/staff understand the data (and comments/specifics) and can they redirect their time and resources?

• Are there improvement targets; is change being measured?


Continuous improvement and assessmentContinuous improvement and assessment

• How will you incorporate the results of your analysis and findings into your Library’s strategic and annual plans?

• How can you develop some of your more resource-intensive improvements over several years?

• How can you insure that your assessment results are always available when and where decisions are being made?

• How and when will you be assessing the user improvements implemented after your user survey?


The importance of accurate data in decision makingThe importance of accurate data in decision making


Your library …Your library …

• Organization and use of information is focused on disciplines and increasingly interdisciplinary centers

• Subject-oriented staff provide information services required and integrate them in the research process

• Developing and maintaining relationships with key stakeholders in the disciplines/departments

• Acquainting specialized stakeholders with the broad range of information and information services available outside their specialty

• Assuring adequate performance of network and technology based information services

• Assuring adequate performance of the centralized basic support functions


ExerciseExercise

• What are the key indicators of this library's work?

• How do you monitor each indicator? How do you record the data for each indicator? With what frequency?

• To whom and how do you communicate information about the library's performance?



Making Library Assessment WorkMaking Library Assessment Work

• ARL project approved in 2004 and funded by participating libraries

• Site visits by Steve Hiller (U of Washington) and Jim Self (U of Virginia)– Presentation– Interviews and meetings – Report to the Library

• 24 libraries in U.S. and Canada visited in 2005-06

• Effective, Sustainable and Practical Library Assessment in 2007– Open to all libraries – 6 libraries participated in 2007


What We FoundWhat We Found

• Strong interest in using assessment to improve customer service and demonstrate value of library

• Many libraries uncertain on how to establish, maintain, and sustain effective assessment

• Effectiveness of assessment program not dependent on library size or budget

• Each library has a unique culture and mission. No “one size fits all” approach works.

• Strong customer-focus and leadership support were keys to developing an effective and sustainable assessment


What are the lessons learned?What are the lessons learned?

• Understanding changes in users approach to information resources.

• Service quality improvement is a key factor.

• Understanding the impact of e-resources on library services - TRL.

• Learning how to compete or collaborate with Google.

• Upfront investment in design and development.

• Making the assessment service affordable, practical, & effective.

• Assessment needs to be satisfying and fun.


Multiple methodsMultiple methods

• Transactional surveys*• Mystery shopping• New, declining, and lost-customer surveys• Focus group interviews• Customer advisory panels• Service reviews• Customer complaint, comment, and inquiry capture• Total market surveys*• Employee field reporting• Employee surveys• Service operating data capture

*A SERVQUAL-type instrument is most suitable for these methods

Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.


Assessment can…Assessment can…

• Support decision making• Establish accountability• Strengthen organizational mission, goals, and objectives• Realign library goals and mission statements with the

needs of users• Identify trends in user behaviors and technology• Measure learning outcomes and impact• Improve programs and services


Use Assessment Data for Good!Use Assessment Data for Good!



What’s in a word?What’s in a word?

What makes a quality

library?

“Quality much like beauty is in the eye of the beholder”


What’s in a “Library”?

A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used.

--Justice Oliver Wendell Holmes


Interpreting DataInterpreting Data

Documents

Why assess? What is assessment? What do we mean by actionable data? Martha Kyrillidou Planning for Results: Making Data Work for You The Cato Institute,