Milestones in Statistical Thinking

Understanding the Methodological Understanding the Methodological Foundations of Public Library Foundations of Public Library

National Rating SystemsNational Rating Systems

Ray LyonsRay Lyons

Library Statistics for the 21st Library Statistics for the 21st Century WorldCentury World

IFLA Satellite Meeting - Montréal, QuébecIFLA Satellite Meeting - Montréal, Québec

August 18 - 19, 2008 August 18 - 19, 2008

Milestones in Statistical Thinking Milestones in Statistical Thinking

• Historical account by Alain DesrosiHistorical account by Alain Desrosièères: res:

The Politics of Large Numbers: A History of The Politics of Large Numbers: A History of Statistical Reasoning Statistical Reasoning (Harvard University Press, 1998) (Harvard University Press, 1998)

• Traces statistical ideas from 15Traces statistical ideas from 15thth century to century to present daypresent day

• Evolutionary ideas from German Evolutionary ideas from German descriptive statistics descriptive statistics and English and English political political arithmetic arithmetic pertinent to library statistical pertinent to library statistical collectioncollection

DesrosiDesrosièères describes two key statistical res describes two key statistical practices:practices:

• Creation of equivalencesCreation of equivalences−Establishing standard classifications to Establishing standard classifications to

describe phenomena by focusing on describe phenomena by focusing on similarities and ignoring differencessimilarities and ignoring differences

• Encoding Encoding −Specification and use of definitions to Specification and use of definitions to

assign individual cases to classificationsassign individual cases to classifications

Milestones in Statistical Thinking Milestones in Statistical Thinking

Purpose of this PresentationPurpose of this Presentation

To suggest that creating equivalent To suggest that creating equivalent classes and other aspects of statistical classes and other aspects of statistical data collection limit the accuracy and data collection limit the accuracy and validity of national and local public library validity of national and local public library comparisons.comparisons.

National Collection of Public Library National Collection of Public Library Statistics in the USAStatistics in the USA

• Federal-State Cooperative System (FSCS)Federal-State Cooperative System (FSCS)

• Collaborative initiated in 1980’sCollaborative initiated in 1980’s

• 1991 - First FSCS data published by U.S. 1991 - First FSCS data published by U.S. Department of Education’s National Center Department of Education’s National Center for Educational Statisticsfor Educational Statistics

National Collection of Public Library National Collection of Public Library Statistics in the USAStatistics in the USA

• Collection system renamed Public Library Collection system renamed Public Library Statistics Cooperative Statistics Cooperative (PLSC) (PLSC) in 2007in 2007

• PLSC recently transferred to Institute of PLSC recently transferred to Institute of Museum and Library Services, an agency Museum and Library Services, an agency of the federal government (www.imls.gov)of the federal government (www.imls.gov)

National Public Library Ratings National Public Library Ratings Introduced in USAIntroduced in USA

• Hennen’s American Public Library RatingsHennen’s American Public Library Ratings

(HAPLR)(HAPLR)

• Used FSCS input and output statistics for Used FSCS input and output statistics for 9000+ public libraries9000+ public libraries

• Created by library consultant Thomas Hennen Created by library consultant Thomas Hennen (library director in Wisconsin)(library director in Wisconsin)

• Issued annually since 1999Issued annually since 1999 (except for 2000 and except for 2000 and 2007)2007)

• Published (and endorsed) by American Library Association (ALA) Published (and endorsed) by American Library Association (ALA)

• Calculation methods were controversial among professionCalculation methods were controversial among profession

• Highly-rated public libraries were delighted with calculation methodsHighly-rated public libraries were delighted with calculation methods

• ““Methodologically indefensible and politically priceless” Methodologically indefensible and politically priceless” - US Library Director- US Library Director

Hennen’s American Public LibraryHennen’s American Public LibraryRatings Ratings (HAPLR)(HAPLR)

• Utilizes 5 enabling (input) indicators and Utilizes 5 enabling (input) indicators and

3 use (output) indicators3 use (output) indicators

Enabling (Input)

Total staffMaterials expendituresTotal operating expendituresNumber of printed volumesSerial subscriptions

Use (Output)

VisitsCirculation (loans)Reference transactions



• Recombines 8 indicators into 15 rates Recombines 8 indicators into 15 rates (ratios), similar in style to BIX(ratios), similar in style to BIX

(1)(1) Total expenditures per capitaTotal expenditures per capita

(2)(2) Materials expenditures per total expenditures Materials expenditures per total expenditures

(3)(3) Materials expenditures per capita Materials expenditures per capita

(4)(4) Staff per 1000 population Staff per 1000 population

(5)(5) Periodical subscriptions per 1000 population Periodical subscriptions per 1000 population

. . . and 10 others. . . and 10 others

• See www.haplr-index.comSee www.haplr-index.com

Study Conducted at NCLIS Study Conducted at NCLIS

• 2006 research on HAPLR ratings 2006 research on HAPLR ratings methodology overseen by U.S. National methodology overseen by U.S. National Commission for Library and Information Commission for Library and Information Science (NCLIS)Science (NCLIS)

“ “Unsettling Scores: An Evaluation of the Hennen Unsettling Scores: An Evaluation of the Hennen American Public Library Ratings, American Public Library Ratings, Public Library Public Library QuarterlyQuarterly, Volume 26, Numbers 3/4, 2007., Volume 26, Numbers 3/4, 2007.

(ISSN 016-6846)(ISSN 016-6846)

Library Journal’s Library Journal’s Public Library Public Library National Ratings AnnouncedNational Ratings Announced

• LJ LJ IndexIndex introduced June 2008 introduced June 2008**

• Rates 9200 public libraries using PLSC dataRates 9200 public libraries using PLSC data

• Co-designed with Keith Lance (Co-designed with Keith Lance (Library Research Library Research

Service,Service, State Library of Colorado)State Library of Colorado)

• Emphasizes disclosing limitations of rating Emphasizes disclosing limitations of rating methodologymethodology

Keith Curry Lance and Ray Lyons, “The New LJ Index,” Keith Curry Lance and Ray Lyons, “The New LJ Index,”

Library Journal, Library Journal, June 15, 2008, p. 38-41.June 15, 2008, p. 38-41.

Library Journal’s Library Journal’s Public Library Public Library National Ratings AnnouncedNational Ratings Announced

• Encourages responsible interpretation of Encourages responsible interpretation of rating resultsrating results

• LJ IndexLJ Index based upon 4 based upon 4 useuse (output) (output) indicators:indicators:

– VisitsVisits

– Loans (circulation)Loans (circulation)

– Internet terminal useInternet terminal use

– Program attendanceProgram attendance

Public Library Ratings UsePublic Library Ratings UseComposite ScoringComposite Scoring

• HAPLR, BIX, and HAPLR, BIX, and LJ Index LJ Index each calculate a each calculate a single composite score summarizing each single composite score summarizing each library’s performancelibrary’s performance

Circulation per capita + FTE staffing +

Subscriptions + Internet computers +

. . . + Total program attendance

Calculation Algorithm

635 (Composite Score)

Context of Comparative Library Context of Comparative Library StatisticsStatistics

• Comparative measures are:Comparative measures are:

− Measures used as part of a more general Measures used as part of a more general process to assess library value and process to assess library value and effectivenesseffectiveness

− Measures intended for use in an ongoing Measures intended for use in an ongoing process of process of performance measurementperformance measurement

Performance Measurement ModelPerformance Measurement Model

ResourcesServices Utilized

Intermediate Outcomes

End Outcomes

Outcome Measures

Outcome Measures

RESULTSEFFORTS

• Planning-for-Results Planning-for-Results approach to approach to management management

• Abandonment of established Abandonment of established operational and performance standardsoperational and performance standards

• ALA / PLA 1987 publication, ALA / PLA 1987 publication, Output Output Measures for Public Libraries: A Manual of Measures for Public Libraries: A Manual of Standardized ProceduresStandardized Procedures, , defines standard defines standard statistics and collection proceduresstatistics and collection procedures

Performance Management as Performance Management as Envisioned by the Envisioned by the

Public Library Association (PLA)Public Library Association (PLA)

Comparative Performance Comparative Performance MeasurementMeasurement

• Public sector management practice Public sector management practice • Used by state and local governments for:Used by state and local governments for:

− AccountabilityAccountability− Planning/budgetingPlanning/budgeting− Program monitoringProgram monitoring− Operational improvementOperational improvement

• Uses established standards and Uses established standards and benchmarking benchmarking (Ammons, 2001)(Ammons, 2001)

• Lack of reliable methods for identifying Lack of reliable methods for identifying peer librariespeer libraries

− Lack of instruments for measuring Lack of instruments for measuring comparabilitycomparability

• Comparisons are either approximate or Comparisons are either approximate or inaccurateinaccurate

• Can result in incorrect or misleading Can result in incorrect or misleading conclusionsconclusions

Key Problems with Library StatisticsKey Problems with Library Statistics

• National rating systemsNational rating systems use simplistic or use simplistic or imprecise criteria for identifying peers imprecise criteria for identifying peers (library type, community size, etc.)(library type, community size, etc.)

• Ignore library mission, unique community Ignore library mission, unique community needs, institutional context, etc.needs, institutional context, etc.

• Accuracy and validity of comparisons are Accuracy and validity of comparisons are compromisedcompromised


• Lack of criteria for evaluating measuresLack of criteria for evaluating measures

There are no ‘right’ or ‘wrong’ scores on an output measure; ‘high’ and ‘low’ values are relative. The scores must be interpreted in terms of library goals, scores on other measures, and a broad range of other factors.

- Van House, Weill, and McClure (1990)


FIG NEWTON COOKIES

Versus

GRANOLA BAR SNACK

100% more fruit!!!

A Diversionary ‘Visual Aid’A Diversionary ‘Visual Aid’

• National rating systemsNational rating systems apply the apply the “ “More-is-Better Rule” More-is-Better Rule”

− Views higher numbers as favorable Views higher numbers as favorable performance, lower as unfavorableperformance, lower as unfavorable

• ““More activity does not necessarily More activity does not necessarily mean better activity”mean better activity”

- Van House, Weill, and McClure (1990)- Van House, Weill, and McClure (1990)

• Striving to earn higher numbers mayStriving to earn higher numbers may compromise service qualitycompromise service quality


• Collection of standard statistics assumes Collection of standard statistics assumes all library resources/activities counted to all library resources/activities counted to be be equivalentequivalent

• Standardization ignores differences in:Standardization ignores differences in:


- Complexity

- Sophistication

- Relevance

- Quality (Merit)

- Value (Worth)

- Effectiveness

- Efficiency

- Significance

• National ratings systems add, subtract, National ratings systems add, subtract, and multiply these non-equivalent ‘units’ and multiply these non-equivalent ‘units’ of library resources, services, and of library resources, services, and products if they were equivalentproducts if they were equivalent

• Final scores imply appear arithmetically Final scores imply appear arithmetically consistent and correct even though they consistent and correct even though they count unequal unitscount unequal units


• Data imprecision due toData imprecision due to

− Inconsistent collection methodsInconsistent collection methods− MistakesMistakes− Sampling errorSampling error− “ “Gaming”Gaming”

• Imprecision makes individual library Imprecision makes individual library comparisons less accuratecomparisons less accurate


• Variety of reasons for insufficient scores:Variety of reasons for insufficient scores:


- Inadequate knowledge of community needs

- Staff skill deficiencies

- Inadequate staffing

- Inefficient workflows

- Inadequate planning

- Limited user competencies

... and others

Adapted from Poll and te Boekhorst (2007)

Output measures “reflect the interaction of Output measures “reflect the interaction of

users and library resources, users and library resources, constrained by constrained by the environment in which they operate.the environment in which they operate. The The meaning of a specific score on any measure meaning of a specific score on any measure depends on a broad range of factors depends on a broad range of factors including the library’s goals, the current including the library’s goals, the current circumstances of the library and its circumstances of the library and its environment, the users, the manner in which environment, the users, the manner in which the measure was constructed, and how the the measure was constructed, and how the data were collected.” data were collected.” [emphasis added][emphasis added]

- Van House, Weill, and - Van House, Weill, and McClure (1990)McClure (1990)

Improvements NeededImprovements Needed

• Fuller understanding of limitations of Fuller understanding of limitations of statistical indicators and comparison statistical indicators and comparison methodsmethods

“ “The [input and output] measures are best The [input and output] measures are best used with other information about the used with other information about the library.”library.”

- Van House, Weill, and McClure (1990)- Van House, Weill, and McClure (1990)

• Relate amounts and types of resources Relate amounts and types of resources and services to and services to verifiable levels of verifiable levels of community needcommunity need

• Increased understanding of Increased understanding of measurement and interpretationmeasurement and interpretation

− Draw Draw reasonable reasonable conclusions and conclusions and interpretationsinterpretations

− Study behavioral science measurement Study behavioral science measurement practicespractices

Improvements NeededImprovements Needed

Behavioral Science Behavioral Science Measurement ModelMeasurement Model

Conceptualization

Nominal Definition

OperationalDefinition

Measurement in Real World

Babbie (2007)

RecommendationsRecommendations

• Promote view of public library national Promote view of public library national rating systems as ‘contests’ with simple, rating systems as ‘contests’ with simple, arbitrary rulesarbitrary rules

• Ratings are:Ratings are:

− Quite approximate indicators of Quite approximate indicators of performanceperformance

− Inadequate indicators of quality, Inadequate indicators of quality, excellence, and effectivenessexcellence, and effectiveness

RecommendationsRecommendations

• Advise libraries to interpret national Advise libraries to interpret national ratings cautiously and conscientiouslyratings cautiously and conscientiously

• Develop and test measurement Develop and test measurement instruments for identifying peer librariesinstruments for identifying peer libraries

• Use ratings to inspire more robust Use ratings to inspire more robust measurement of public library measurement of public library performanceperformance

Documents

Milestones in Statistical Thinking