LibQUAL+ Canada Consortium Survey Results Webinar

Oct. 5, 2010

Organized by Sam Kalb, LibQUAL+ Canada Coordinator

Sponsored by the Canadian Association of Research Libraries

Moderated by Katherine McColgan, CARL

Program

1. Sam Kalb (Queen’s University). Using the LibQUAL +® notebooks and other LibQUAL+ services to analyze and present your results (50 minutes)

2. Eun-ha Hong (Wilfrid Laurier University). Using statistical tools to further analyze your LibQUAL+® data (15 minutes)

3. Questions/discussion (25 minutes)

Understanding & Using Your LibQUAL+®

Results

Sam KalbAssessment & Scholarly Communication Services

CoordinatorQueen’s University Library

Privacy & LibQUAL +® Results

• Any LibQUAL participant can access the results of all other libraries and consortia who participated in the same year

• “In an example of collaboration, LibQUAL+® participants are sharing their results within the LibQUAL+® community with an openness that nevertheless respects the confidentiality of each institution and its users.” Martha Kyrillidou, 2010 LibQUAL +® survey, introduction.

Desired Outcomes

• Use the LibQUAL +® Notebooks & analytic utilities to perform some simple analyses of your LibQUAL+® survey results

• Present the results to your stakeholders

• Use the data to target areas for improvement

From: presentation by R. Bowlby and M. Kyrillidou,, LibQUAL+® Canada Workshop, Ottawa, Ontario, Canada, October 24-25, 2007

LibQUAL+ Canada 2010 Survey Format Distribution

Total LibQUAL+® Canada Members 2010 Full Lite Total

Universities Count 11 33 43*

% of Protocol 84.6 91.7

% of Total Cases 23.4 70.2 91.5*

Community Colleges Count 2 3 4*

% of Protocol 15.4 8.3

% of Total Cases 4.3 6.4 8.5*

Total (by Survey Protocol)

Count 13 36 47*

% of Protocol 100 100

% of Total Cases 27.7 76.6 100*

¾ of members chose the Lite format

Completion Rates

• 2010 LibQUAL+® Lite: 61.7%

• 2010 LibQUAL+® full: 54.3%

• 2007 LibQUAL+® full: 48.8%

Completed Survey. The user has supplied a rating for all items on the survey.

Relative Indicators

1. There are no absolute high or low scores. 2. Scores are relative indicators3. Scores are only meaningful in comparison with

other scores in the same survey, your survey from another year, other individual libraries and the consortial totals.

Three interpretation frameworks1. Zone of tolerance

• Perceptions vs. expectations• meeting users minimum expectations (Adequacy Gap)• Approaching users’ desired expectations (Superiority

Gap)

2. My scores over time (longitudinal)• Am I doing better or worse compared to last time I

measured my performance

3. Peer comparisonsFrom presentation by M. Kyrillidou, ALA, June 2007

What Do the Ratings Signify?

User Assigned:• Desired. How highly do I value it?• Minimum. What is my least expectation?• Perceived. My actual rating

Calculated Scores:• Adequacy Gap. Perceived – Minimum• Superiority Gap. Desired – Perceived

Which Numbers?

1. Desired. How important is a service relative to other services; relatively to same services among other libraries

2. Adequacy Gap. How far above or below Minimum expectation is the service delivered.

User Groups & Disciplines

User GroupsFocus mainly on your results by User Group

The major academic user groups (faculty, grads and undergrads) have quite different perspectives and expectations of library services.

Overall totals from different libraries or different years could be skewed by variations in user groups ratios. • Look at the 5 most and the 5 least desired questions by each

user group• Look at the services with 5 highest and the 5 lowest gap scores

(Bear in mind that Superiority Gap scores are usually negatives.• Look for any correlation between the high/low Desired and the

high/low Gap scores• Look at the average (mean) scores for each service dimension

by user group

5 highest Desired Services

Queen's 2007 Queen’s 2010 Canadian Consortium (Universities) 2010

Dim. Question Score Dim. Question Score Dim. Question Score

Overall Overall Overall

IC-1 Making electronic resources accessible from my home or office

8.51 

IC-1 Making electronic resources accessible from my home or office

8.51 IC-1 Making electronic resources accessible from my home or office

8.22

IC-8 Print and/or electronic journal collections I require for my work

8.48 

IC-2 A library Web site enabling me to locate information on my own

8.38 IC02 A library Web site enabling me to locate information on my own

8.17

IC-4 The electronic information resources I need

8.39 

IC-8 Print and/or electronic journal collections I require for my work

8.32 IC-8 Print and/or electronic journal collections I require for my work

8.16

IC-2 A library Web site enabling me to locate information on my own

8.38 

IC-4 The electronic information resources I need

8.16 IC-6 Easy-to-use access tools that allow me to find things on my own

8.06

IC-6 Easy-to-use access tools that allow me to find things on my own

8.30 IC-6 Easy-to-use access tools that allow me to find things on my own

8.15 AS-3 Employees who are consistently courteous

8.06

Academic Discipline Groups

• In addition to analyzing your results by User Group, you can analyze your results by Discipline using ARL’s Analytics utility (to be covered later in the presentation).

• Focus on LibQUAL +®’s Standard Disciplines › Facilitates comparison with other libraries› Most useful for Information Control dimension

Where to Get Your Results

• Report - results notebook (PDF)• Comments - respondents’ free-text comments (CSV or

TXT)• Consortial Notebooks – Consortium, CARL, CREPUQ,

OCUL (PDF)• Raw Data – individual records of every respondent’s

survey in CSV format (spreadsheet)• Data Keys & SPSS Syntax – description of the raw

data elements or fields (MS Word)http://www.libqual.org/SurveyInstruments/LibQual/DataRepository.aspx

Consortial Results Notebooks

http://library.queensu.ca/webir/canlibqual/results-e.htm

The Results Notebooks

Results Notebooks - Content

• Sections for Overall, Undergraduates, Graduates, Faculty, Staff, Library Staff include:› Demographic Summary› Core Questions Summary› Dimensions Summary› Local Questions› General Satisfaction Questions› Information Literacy Outcomes Questions› Library Use Summary

• Appendix describing changes in the dimensions and the questions included in each dimension.

Representativenessby User Group &

Discipline

Representativeness - Standard Disciplines: An Academic Library*

* Queen’s University 2010 LibQUAL +® Survey

LibQUAL+ Canada 2010University Response Rates

Population Respondents Response Rate

Undergrads 473,406 30,173 6.7%Grads 104,867 11,201 10.7%All Faculty 47,867 51,45 10.7%Full-time Faculty 29,039 4,606 15.9%

Core Survey Questions• Tables & Charts

• Individual questions – average scores & standard deviations

• Dimensions summary

Standard Deviations[This slide was omitted from Oct. 5/2010 presentation]

How closely does a mean score in a notebook represent all the individual respondents scores for the particular item. If all respondents rated AS-1 Desired as 7.64, the SD would be 0

Understanding Your Thermometer Charts

Desired

Perceived

Minimum

Zone of tolerance

Target

Charting the Survey Dimensions

Range of Mean scores is relatively

narrow(6.27 – 8.03)

• Affect of Service – Customer/client service• Information Control – Collections & access to

collections• Library as Place – Physical facilities

LibQUAL Canada 2010 Faculty Mean Scores

Negative Adequacy Gap Scores • Potential areas for improvement or further

investigation

When lower may not be a bad sign?

Survey Year

MinimumMean

PerceivedMean

DesiredMean

Adequacy Gap

current 6.72 7.45 7.85 0.73previous 6.33 7.29 7.80 0.96

Always check the Minimum mean score when evaluating the rise or fall in your Adequacy Gap score. A lower Gap score just may result from rising expectations

Understanding Your Individual Results: Radar Charts

From presentation by M. Kyrillidou, ALA, June 2007

Key to Radar Charts

From presentation by M. Kyrillidou, ALA, June 2007

Other Notebook Elements

• Local Questions• General Satisfaction Questions• Information Literacy Outcomes

Questions• Library Use Summary

Local Questions

LibQUAL Canada 2010. Item 3.4. Location Questions Summary

• You can compare your individual local question results with:• Your past surveys• Consortial or peer library results (assuming

sufficient number of respondents, e.g. 100+)

General Satisfaction &Information Literacy Outcomes Questions

• These are simple indicators without the context of minimum and desired ratings

LibQUAL+ 2010 – General Satisfaction  LibQUAL+

Canada Queen's

In general, I am satisfied with the way in which I am treated at the library. 7.54 7.59In general, I am satisfied with library support for my learning, research, and/or teaching needs. 7.11 7.14How would you rate the overall quality of the service provided by the library? 7.19 7.32

Library Use Summary

From presentation by M. Kyrillidou, ALA, June 2007

Qualitative Analysis: User Comments

• Almost ½ of respondents fill in the Comments box

• Provide valuable insights and suggestions for improvements

• Essential component in understanding the reasons behind the survey scores.

• Available on LibQUAL+® Web site Data Repository

www.libqual.org

Comments Records Download file in csv(Excel) or txt (text) Record Content:

› ID:› UserGroup:› Discipline: › Branch:› Age: › Sex: › Comment:

Using Respondents’ Comments• Skim the comments• Load csv file into text analysis program, e.g.

ATLAS.ti, InMagic, etc. • http://db.library.queensu.ca/libqual/index2010.htm

• Conduct analysis• Identify major themes, e.g. study space,

library catalogue, noise, etc.› http://library.queensu.ca/webir/libqual-2007/issues&actions.html

Other LibQUAL+® Analytical Tools

• Analytics

• Norms Tables

www.libqual.org

LibQUAL+® AnalyticsUser group and Discipline analysis

• Institutional Explorer (peer comparison)•Representativeness graphs•Radar chart•Library Use chart•Thermometer chart•Cumulative percentile distribution

• Longitudinal Analysis (by survey year)• Cumulative percentile distribution

http://www.libqual.org/SurveyInstruments/LibQual/Analytics.aspx

LibQUAL +® AnalyticsInstitutional Explorer

2010 Queen’s LibQUAL +® resultsUndergraduate Business students

Longitudinal AnalysisQueen’s University, 2007 & 2010

Heath Sciences Graduate Students

LibQUAL +® Chart Templates

• ARL provides templates to assist you in preparing customized radar and thermometer charts to create custom analyses

www.libqual.org

http://www.libqual.org/SurveyInstruments/LibQual/Resources.aspx

LibQUAL+® Norms Tables• Identify your score

• Compare it to a relevant norms table• Year

• Subgroup

• Dimension

• Norms are stable and are not calculated on an annual basis anymore (last calculated 2005)

• Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample.

http://people.cehd.tamu.edu/~bthompson/libq2005.htmFrom: presentation by R. Bowlby and M. Kyrillidou,, LibQUAL+® Canada Workshop, Ottawa, Ontario, Canada,

October 24-25, 2007

Additional LibQUAL +® Analysis Services

• Customized Discipline Analysis• Library Branch Analysis• Other Customized Analyses (upon

request)• Print Copies

http://www.libqual.org/about/about_lq/fee_schedule

www.libqual.org

Presenting the Results to Your Stakeholders

Stakeholders

• Identify all of the stakeholders or constituents who want and need to know about the survey results

• Consider the “stake” of each of the above; what specific aspect of LibQUAL+® will be of most interest / concern

• Determine how to communicate with each identified stakeholderFrom: presentation by R. Bowlby and M. Kyrillidou,, LibQUAL+® Canada Workshop, Ottawa, Ontario, Canada, October

24-25, 2007

Communicate with your Customers(faculty, students, other)

1st Priority. Particularly those whom you asked to participate in the survey. As soon as you can:1. Announce incentive award winners2. Inform users of highlights of survey results3. Present weak areas as challenges and

opportunities not as negatives4. Most importantly, what the library intends to do.

Describe action items begun and planned

Example: http://library.queensu.ca/libqual%202010From: presentation by R. Bowlby and M. Kyrillidou,, LibQUAL+® Canada Workshop, Ottawa, Ontario, Canada, October

24-25, 2007

Comparing Your Library’s Results

• Compare your results with the corresponding consortial results and those of peer Canadian libraries

• Compare your results over multiple LibQUAL +® surveys (longitudinal analysis)

• Look at results to determine if users are not aware of what the library already does

• Explore one question by discipline and user group• Probe the questions that had meaningful gaps between

perceived results and minimum expectations (Adequacy Gap)

From presentation by M. Kyrillidou, ALA, June 2007

Targeting Incremental Improvements

• From all of the data, determine what can and should be addressed

• Prioritize some action items• Align with mission, vision and goals of parent

organization• Address users’ top priorities, by user group• Improve areas of strong user dissatisfaction• Build on strengths, if they are truly user needs and

priorities• Identify work that can be de-emphasized and

resources that can be reallocatedFrom: presentation by R. Bowlby and M. Kyrillidou,, LibQUAL+® Canada Workshop, Ottawa, Ontario, Canada, October

24-25, 2007

What should go into a report

• Carefully crafted Executive Summary

• Library performance (strengths & opportunities for improvement/ enhancements in services and priorities

• Present results via explanatory text with supporting tables and charts in an appendix. Don’t rely on tables and chart to convey your point.

Example from Library Web Site: http://library.queensu.ca/webir/libqual-2007/issues&actions.html

2007 Findings Actions

Prepare Action Plans

Peer Comparisons

Queen’s University LibQUAL+ 2010 - Core QuestionsSummary by User Group

User Group Minimum (Mean/Avg)

Desired (Mean/Avg)

Perceived (Mean/Avg)

Adequacy Gap

Queen's CanCon Queen's CanCon Queen's CanCon Queen's CanCon

Faculty 6.55 6.75 7.72 7.83 7.18 7.03 0.63 0.28

Graduate 6.62 6.67 7.93 7.97 7.11 7.04 0.49 0.37

Undergraduate 6.27 6.47 7.83 7.87 6.89 6.96 0.62 0.49

Example: Queen’s University 2010 user group scores compared with the LibQUAL Canada Consortial results

A Sample Plan for Presenting Your Completed Analysis

Queen’s University Library – 2007 SurveyResults Presentation Plan

• Completed Report discussed at Management Team; consultation plan developed

• Report, comments database, and plan distributed to all staff• All-Staff information session• Units and functional teams• Meetings of individual units and functional teams identify the

issues in their areas of responsibilities and recommend appropriate actions.

• Management Team• Reviewed the compilation of issues and objectives in

developing the 2008/09 Budget Report. Compiled and approved action items prepared by the functional teams and units.

Overcoming Resistance & Barriers

old.libqual.org

• Engage the staff in reviewing the results and developing solutions

• Focus on assessment of services not people; don’t make the survey a staff assessment tool

• Keep the experience a positive one for the staff

Service Quality Evaluation Academy

• The academy is an intensive five-day program that focuses on both qualitative and quantitative methods for collecting and analyzing library service quality data.

• Next scheduled for March 14-18, 2011, Toronto, Ont.

• Deadline for application: December 15, 2010

http://www.arl.org/stats/statsevents/sqacademy/index.shtml

LibQUAL+® Resources

• LibQUAL Canada: http://library.queensu.ca/webir/canlibqual/carl-libqual.htm

• LibQUAL +®: www.libqual.org• Other library web sites

• http://library.queensu.ca/webir/canlibqual/canlibs.htm

• http://www.libqual.org/about/about_survey/related_sites

Q & A