LibQUAL+®, MINES for Libraries™ and Other Tools Packaging Results to Communicate to Stakeholders Planning for Results: Making Data Work for You The Cato

LibQUAL+®, MINES for Libraries™ and Other Tools Packaging Results to Communicate to Stakeholders

Planning for Results: Making Data Work for YouThe Cato Institute, Washington, DC

February 20, 2008

Gary Roebuck, Technical Operations ManagerKristina Justh, Customer Relations Coordinator

Statistics and Measurement ProgramAssociation of Research Libraries

Where we’ve been...a recap

The Assessment Process…

Initiate (Why do we assess?)

Plan (What do we assess?)

Design(How do we assess?)

Implement(Design in action)

Analyze and Interpret(What do the results mean?)

Reflect and Initiate (What do we do with the results?)

“Why assess? What is assessment? What do we mean by actionable data?” - Martha Kyrillidou

“Survey Design” – Neal Kaske

“Performance Measures” – Rick Garcia

“Analysis and Interpretation” - Ray Lyons

What’s left?

The Assessment Process…

Initiate (Why do we assess?)

Plan (What do we assess?)

Design(How do we assess?)

Implement(Design in action)

Analyze and Interpret(What do the results mean?)

Reflect and Initiate (What do we do with the results?)

LibQUAL+®, MINES™, and Other Tools – examples of assessment tools developed through partnerships of ARL with other researchers

Packaging Results to Communicate to Stakeholders; and Assessment as a PR Tool –Communicating the results and forthcoming actions in a compelling and public relations enhancing manner

But first…

Staff Processes

Tools Product

Users Processes

Technology Data

Information System Model Generalized Organization Model

What should we assess? Consider this framework…

A Generalized Organization Model

Staff Processes

Tools Product

Customers

• Staff – puts processes and tools to use to produce, enhance, or disseminate product

• Processes – details how functions of the organization are performed

• Tools – includes technical, financial, and physical resources used to carry out the organization’s mission

• Product – books, databases, e-materials, etc.

What should we assess?

Sample Quantitative• Average staff tenure

• Number of reference transactions filled

Sample Qualitative• How do staff view/feel about work

environment, management?

• How are staff perceived by users…are they friendly, resourceful, knowledgeable?

Staff


Sample Quantitative• How long (on average) does an inter-

library loan request take?

• How many steps does it take to prepare a new resource for usage?

Sample Qualitative• How simple/convenient is it to make an

inter-library loan request?

• Does the material preparation process delay materials for which users are currently waiting? How does this impact their research, teaching, and/or learning?

Processes


Sample Quantitative• How many hours is the library open?

• What is the budget for electronic materials?

• How many hits does the library web portal get in a given month?

Sample Qualitative• Are the hours the library is open meeting

the needs of users?

• Are the investments in electronic materials deemed adequate by faculty and students?

• Is the library’s web portal/search tool perceived as user friendly and effective?

Tools


Sample Quantitative• How many unique serial titles are owned

by the library?

• How many electronic journals does the library subscribe to?

Sample Qualitative• Do the students and faculty feel the

selection of titles provided by the library serves their needs?

• Does the library have the journal titles needed by the faculty and students?

Product

ARL Tools for Library Assessment

In the beginning…

ARL Statistics™

ARL Statistics is a series of annual publications that describe the collections, expenditures, staffing, and service activities for ARL member libraries. These data have been collected since 1907-08, initially by James Gerould. Since 1961-62 ARL has collected and published these data annually. The whole data series represents the oldest and most comprehensive continuing library statistical series in North America.

The series includes: • ARL Statistics • Academic Health Sciences Library Statistics • Academic Law Library Statistics• ARL Preservation Statistics• ARL Supplementary Statistics• Annual Salary Survey


As a result of the work of the New Measures and Assessment Initiative (1999)…

ARL Statistics™

Since 1907-08

LibQUAL+®

Since 2000

MINES for Libraries™

Since 2003

DigiQUAL®

Since 2003

ClimateQUAL™

Since 2007


As we look to the future…

New Tool New

Tool

New Tool

Common Database

StatsQUAL®

• Common User Interface

• Unified Data Structure

• Enhanced Data Mining/Warehousing Opportunities

• Faster Development Cycle for New Tools

•Common Workflow

Users


Tool Staff Processes Tools Product

ARL Statistics™ X X X X

LibQUAL+® X X X

ClimateQUAL™ X


X X

DigiQUAL® X X

What the tools assess…

LibQUAL+®

• Suite of services that libraries use to solicit, track, understand, and act upon users’ opinions of service quality

• Administered via Web interface; “22 questions and a box”

• 1000+ libraries around the world

• Allows for standardization across institution types and benchmarking capabilities

• http://www.libqual.org

http://www.libqual.org/

LibQUAL+®

ClimateQUAL™

• Organizational Climate and Diversity Assessment

• 200+ question, Web-based survey of library staff’s perception of organizational climate

• Results help develop a “healthy organization” -- one in which consistent messages regarding its policies and procedures are provided to all employees

• Connection between organizational procedures/policies and customer perception of service quality


•Measuring the Impact of Networked Electronic Resources (MINES for Libraries™)

•5 question, Web-based survey

•Administered at the time access

•Collects data on •user status•disciplinary affiliation•physical location at time of access•primary purpose of use•reason for selecting resource

DigiQUAL®

•Modeled after LibQUAL+® protocol

•Assesses the services provided by digital libraries

•Questions centered on 12 themes, e.g.:•design features•accessibility/navigability•collection building•copyright

•Libraries select survey administration method

Packaging Results to

Communicate to Stakeholders

________________________________________

Assessment as a PR Tool

The data is analyzed…now what?

Need more data?Collect more data

(focus groups, targeted surveys, individual

interviews, etc)

Set and implement action plan

Publicize findings and actions

Yes

No

Publicize findings and actions…

Public Relations 101• Who? Who are your audiences?

• What? What are your messages?

• Where? What location do you choose to deliver your message?

• When? Who do you communicate with first? Second? Third?

• How? What mode of communication do you use…in-person event, e-mail, staff

meeting, etc?

A focus on the What and How…

Delivering a quantitative message…

• General principles

• Types of quantitative messages (and how to best present them)

• Evaluating the presentation of a quantitative message?

Delivering a quantitative message...

Cleveland, William S. 1994. The Elements of Graphing Data, rev. ed. Hobart Press, Summit, NJ

Few, Stephen. 2004. Common Mistakes in Data Presentation. Retrieved February 10, 2008 from http://

www.intelligententerprise.com/showArticle.jhtml?articleID=26100530&pgno=1

Few, Stephen. 2004. Eenie, Minie, Minie, Moe: Selecting the Right Graph for Your Message. Retrieved February 10, 2008 from http://www.intelligententerprise.com/showArticle.jhtml;jsessionid=RYGRFUJY5PETWQSNDLPSKH0CJUNN2JVN?articleID=46800517

Robbins, Naomi B. 2005. Creating More Effective Graphs. John Wiley and Sons Inc., Hoboken, NJ

Tufte, Edward R. 2001. The Visual Display of Quantitative Information, 2nd ed. Graphics Press, Cheshire, CT

http://www.intelligententerprise.com/showArticle.jhtml?articleID=26100530&pgno=1



http://www.intelligententerprise.com/showArticle.jhtml;jsessionid=RYGRFUJY5PETWQSNDLPSKH0CJUNN2JVN?articleID=46800517

http://www.intelligententerprise.com/showArticle.jhtml;jsessionid=RYGRFUJY5PETWQSNDLPSKH0CJUNN2JVN?articleID=46800517

General Principles

(Few, 2004)

1. Start with a clear message

2. Don’t insist on a graph

3. Use the right type of graph

4. Express don’t impress

“In anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away”

Antoine de St. Exupery

General Principles

1. Start with a clear message

• “Before you can determine how to effectively present a message you must first know what the message is.”

• “Actually put [the message] into one or more sentences that are complete and meaningful…You can find clues in the words you use to express your message that will direct you to present data in a particular way.”

(Few, 2004)

General Principles

2. Don’t insist on a graph

Basic Structure for showing data (Tufte, 2001)

• The Sentence – Best unless you are trying to compare more than two numbers with a data set.

• The Table – Ideal for small data sets and instances where you are trying to show exact numerical values

• The Graph – Ideal when trying to communicate a message that is contained in the shape of the data or to reveal relationships among many values (Few, 2004)

General Principles

3. Use the right type of graph

• More on this in the next section (types of quantitative messages)

• A word about pie charts…

Pie charts require judgments on angles. As humans, “we underestimate acute angles and overestimate obtuse angles” (Robbins, 2005)

“The only thing worse than a pie chart is several of them…Given their low data density and failure to order numbers along a visual dimension, pie charts should never be used.” (Tufte, 2001)

General Principles


(Cleveland, 1994)• Make the data stand out. Avoid superfluity (“Chartjunk”)

• Use visually prominent graphical elements to show the data

• Do not clutter the interior of the scale-line rectangle

• Visual clarity must be preserved under reduction and reproduction

• Proofread graphs

General Principles


(Robbins, 2005)• Do not overdo the number of tick mark labels

• De-emphasize grid lines and distinguish grid lines from data

• The chart or graph must be consistent with the text

• Draw data to scale

General Principles

Some final words …

(Robbins, 2005)• A large amount of quantitative data can be packed into a

small region

• Graphing data should be an iterative, experimental process

• Graph data two or more times when needed

• Many useful displays [still] require careful, detailed study

Types of Quantitative Messages (Few, 2004)

Vocabulary

Data• Quantitative – values that represent measures of things (How much?)

• Categorical – describes what a quantitative data measures (What?)– Nominal (labels that have no intrinsic order)– Ordinal (labels that have an intrinsic order)– Interval (labels created from the conversion of a quantitative scale)

Pre-attentive attributes (items that combine to form perceived objects)

•2-D Location •Line Length

•Size •Shape

•Orientation •Color


Three types of objects to encode quantitative values

• Points – can take the form of any geometric shape. Visually emphasizes individual values

• Lines – used to (1) connect a series of individual data points and/or (2) display the trend of a series of data points

• Bars – combines both attributes to emphasize individual values and support comparison of one to another.

0

10

20

0 1 2 3 4 5

0

10

20

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr0

10

20

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr


1. Nominal Comparison

2. Time Series

3. Ranking

4. Part-to-Whole

5. Deviation

6. Frequency Distribution

7. Correlation

Just about every quantitative message you’ll ever need can be described as one or a combination of the seven following types:


Semester Averages

0

20

40

60

80

100

John

Susie

Tom Jane

Students

Per

cen

tag

e

1. Nominal ComparisonA simple comparison of the categorical subdivisions of one or more measures in no particular order.

Encoding Methods• Bars only (horizontal or vertical)


John's Semester Average - by month

0

20

40

60

80

100

Jan

Feb Mar Apr

May

June

Month

Per

cen

tag

e

2. Time SeriesMultiple instances of one or more measures taken at equidistant points in time

Encoding Methods• Lines to emphasize overall patterns

• Bars to emphasize individual values

• Points connected by line to emphasize individual values while highlighting overall pattern


Semester Averages

0 20 40 60 80 100

John

Susie

Jane

Tom

Stu

den

ts

Percentage

3. RankingCategorical subdivisions of a measure ordered by size (either descending or ascending)


• To highlight high values, sort descending

• To highlight low values, sort ascending


Grade Composition

0%10%20%30%40%50%60%70%

Ho

me

wo

rk

Ex

am

s

Te

rm P

ap

er

Pa

rtic

ipa

tio

n

Component

Pe

rce

nta

ge

4. Parts-to-WholeMeasures of individual categorical subdivisions as ratios to the whole.


• Use stacked bars only when you must display measures of the whole as well as the parts

Types of Quantitative Messages (Few, 2004)5. DeviationCategorical subdivisions of a measure compared to a reference measure, expressed as the differences between them.

Encoding Methods• When displaying deviation and time-series relationships together:

• Lines to emphasize the overall pattern

• Points connected by lines to emphasize individual data points while also highlighting the overall pattern

• Bars to emphasize individual values (limit to vertical bars when a time-series relationship is included)

Semester Averages to Class Average Variance

65

70

75

80

85

90

95

John Susie Tom Jane

Students

Pe

rce

nta

ge


# of Students by Semester Average

0

5

10

15

20

<60 >=60& <70

>=70& <80

>=80& <90

>=90

Semester Average

# o

f S

tud

ents

6. Frequency Distribution

Counts of something per categorical subdivisions (intervals) of a quantitative range.

Encoding Methods•Vertical bars to emphasize individual values (called a histogram)

•Lines to emphasize the overall pattern (called a frequency polygon)


7. CorrelationComparisons of two paired sets of measures to determine if as one set goes up the other set goes either up or down in a corresponding manner, and if so, how strongly.

Correlation of Semester Averages and # of Homework Assignments Missed

0

20

40

60

80

100

120

0 2 4 6 8 10 12 14 16

# of Assignments Missed

Sem

este

r A

vera

ge

Encoding Methods• Points and a trend line in the form of a scatter plot

• Bars may be used, arranged as a paired bar graph or a correlation bar graph, if scatter plots are unfamiliar

Evaluating the presentation of a quantitative message? (Tufte, 2001)Graphical displays should:

1. Show the data

2. Induce the viewer to think about the substance

3. Avoid distorting what the data have to say

4. Present many numbers in a small space

5. Make large data sets coherent

6. Encourage the eye to compare different pieces of data

7. Reveal the data at several levels of details, from a broad overview to the fine structure

8. Serve a reasonably clear purpose: description, exploration, tabulation, or decoration

9. Be closely integrated with the statistical and verbal descriptions of a data set

Communicate, Market, Share

• Marketing is part of overall communication strategy

• Visual representations of data using charts or graphs

• In person events: e.g., Library Summit, focus groups

• Permanent library Web site with data, analysis, and follow-up

Visual Representations of Data

Visual Representation of Data


9

5

7

8

4

6


Source: Brian Mathews, Georgia Tech, ALA Annual Meeting, Washington, DC, June 25, 2007


MED

NURS

SAS-Sci

SAS-SSci

ENGNR

VETSAS-H

um

WHRTN

DENTAL

SOC WRK

EDUCOM

MN

Total

Perce

nt

Health Science 11,101 1,736 304 340 55 251 59 45 249 161 42 55 14,398 66.55%

Chemistry 2,193 195 334 61 158 54 6 5 68 9 2 0 3,085 14.26%

Arts & Humanities 323 87 39 124 13 27 228 26 4 68 177 29 1,145 5.29%

Engineering 437 10 49 21 89 7 7 11 8 0 0 0 639 2.95%

Business & Econ 101 50 11 38 20 0 17 249 0 15 12 12 525 2.43%

Agriculture 202 16 131 11 3 69 0 0 4 0 1 0 437 2.02%

Library & Info Sci 288 4 50 12 4 6 0 0 7 0 0 0 371 1.71%

Physics 54 8 14 26 13 0 31 8 0 38 34 12 238 1.10%

Enivronmental Sci 130 21 34 6 5 10 2 0 4 6 0 2 220 1.02%

Life Science 107 2 38 1 5 3 0 0 4 0 0 0 160 0.74%

Social Sciences 20 5 1 17 3 1 45 13 0 17 19 6 147 0.68%

Info Tech 21 1 23 9 17 1 2 3 0 0 0 0 77 0.36%

Material Science 3 0 25 0 27 0 0 0 0 0 0 0 55 0.25%

Math Science 9 0 10 0 34 0 0 2 0 0 0 0 55 0.25%

Law 6 0 1 5 0 0 20 1 0 9 2 0 44 0.20%

Earth Science 4 0 27 2 4 0 0 1 0 0 0 0 38 0.18%

14,999 2,135 1,091 673 450 429 417 364 348 323 289 116 21,634 100.00%

69.33% 9.87% 5.04% 3.11% 2.08% 1.98% 1.93% 1.68% 1.61% 1.49% 1.34% 0.54% 100.00%

Faculty Use of Electronic Journals: School by Journal Topic [21,634 article views]

Source: Joe Zucca, University of Pennsylvania, Library Assessment Forum, ALA Midwinter, January 11, 2008

Peer ComparisonsLibQUAL+ 2006

Faculty Ratings of Journal CollectionsARL Libraries

5.00

6.00

7.00

8.00

9.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

Red Square = Perceived Service Performance

Top of Blue Bar = Desired Level of Service

Bottom of Blue Bar = Minimum Level of Service

UVA

Source: Jim Self, University of Virginia, Presented at Performance Measurement in Academic Libraries Workshop, EBLIP4, Durham, North Carolina, May 11, 2007

http://www.libqual.org/documents/admin/EBLIP4.ppt

Peer Comparisons

Source: Brian Mathews, Georgia Tech, ALA Annual Meeting, Washington, DC, June 25, 2007

9

5

7

8

4

6

3

Peer Comparisons

Adequacy GapThe difference between the minimum and perceived score.

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

UT Austin

ARL

Peers

Source: Fred Heath, LibQUAL+® Results Meeting, ALA Annual Conference, Seattle, WA, January 22, 2007

Peer Comparisons

Source: Jim Self, University of Virginia, Presented at Performance Measurement in Academic Libraries Workshop, EBLIP4, Durham, North Carolina, May 11, 2007

http://www.libqual.org/documents/admin/EBLIP4.ppt

Library Summit

Library Summit

•Gather stakeholders•Opportunity to share survey results•Facilitated small group discussions on:

–issues raised by survey results–solutions/suggestions for improvement

•Major visibility for library•Chance to market library resources & services•Catalyst for constructive dialog and institution-wide collaboration

Organizational Memory

Access to Data

Organizational Memory - Access to Data

http://library.uiuc.edu/assessment/


http://www.lib.clemson.edu/aboutlib/overview/plans.htm


http://ucblibraries.colorado.edu/internal/assessment/datasets.htm


http://metrics.library.upenn.edu/prototype/datafarm/

Documents

LibQUAL+®, MINES for Libraries™ and Other Tools Packaging Results to Communicate to Stakeholders Planning for Results: Making Data Work for You The Cato