Impact measuresfor libraries and

information services

Roswitha PollMünster

Bielefeld2006

Bielefeld2006

The importance of libraries

What libraries ensure (IFLA)

Democraticvalues

Economicprogress

The open information

society

Information equality

Reduction of poverty

Information literacy

Intellectual freedom

Culturaldiversity

Libraries make a differenceBut who will believe it?

Bielefeld2006

Bielefeld2006

inputinput

activitiesactivities

impactimpact

outcomeoutcome

valuevalue

outcome

a contribution of work, information, or materiala contribution of work, information, or material

activities change input into outputactivities change input into output

the effect or influence of one person, thing, orthe effect or influence of one person, thing, oraction, on anotheraction, on another

the consequence, visible or practical result orthe consequence, visible or practical result oreffect of an event or activityeffect of an event or activity

the importance or preciousness of something,the importance or preciousness of something,the perception of actual or potential benefitthe perception of actual or potential benefit

outputoutput the amount produced, the results suppliedthe amount produced, the results supplied

Bielefeld2006

Bielefeld2006

Don Revill

ACRL

Bielefeld2006

Bielefeld2006

Don Revill

ACRL

Peter Brophy

Different outcomes

positive — negative

direct/immediate — long-term

actual — potential

intended — unexpected

Bielefeld2006

Bielefeld2006

Impact/outcome of cultural institutions

knowledge

information literacy

academic or professional success

social inclusion

individual well-being

changes in skills, competences,

attitudes, behaviour

Bielefeld2006

Bielefeld2006

Outcomes of universities

recruitment and retention of students and excellent academic staff

effective teaching - high graduation rates - high grades in examinations - high employment rates after examination

effective research- high renown and use of research results and publications - high amount of special grants - awards, honours

Evidence needed:The library supportsthese goals

Bielefeld2006

Bielefeld2006

Problems of assessing outcome

Value and outcome of a service can differ between user groups

Data may not be available because of data protection rules

Results of projects are often not comparable because of different data collection methods

Long-term effects cannot be assessed as persons are no more available

All tested methods are time-consuming

Influences on individuals are

manifold

Bielefeld2006

Bielefeld2006

Methods for assessing impact

Quantitative measures

tests assessing user skills before and after training on a service

performance monitoring / data mining

observation (unobtrusive)

analysis of documents and bibliographies compiled by users

correlation between library use and success

Bielefeld2006

Bielefeld2006

"Surrogate measures" have

to be used

Methods for assessing impact

Qualitative measures

Surveys (print, telephone, online)

Interviews

Focus groups, discussion groups

Self-assessment of users

Facts and stories must be organized to show patterns

The "anecdotal evidence"

supports data

Bielefeld2006

Bielefeld2006

ACRL

Jennifer Cram

User satisfaction as outcome measure?

Bielefeld2006

Bielefeld2006

A change of behaviour?

Or a basis forsuch change?

Correlation of library useand academic success

Frequency (loans, visits)Frequency (loans, visits)Range of services used (reference service, user training, ILL)Range of services used (reference service, user training, ILL)Range of collections used (undergraduate collection, E-journals, special Range of collections used (undergraduate collection, E-journals, special material) material) Attendance at training sessionsAttendance at training sessions

Success, expressed bySuccess, expressed byDuration of studiesDuration of studiesGrades in examsGrades in examsStudent retentionStudent retentionEmployment rate after examEmployment rate after examQuality of publications (impact factor,Quality of publications (impact factor,peer-review journals)peer-review journals)

Library use, expressed byLibrary use, expressed by

Assessing impact

Bielefeld2006

Bielefeld2006

The library's impact on information literacy

• surveys after instructionsurveys after instruction

• pretest / posttestpretest / posttest

• self-assessment of usersself-assessment of users

• behavioural observationbehavioural observation

• transaction logstransaction logs

• longitudinal studieslongitudinal studies

• assessing changes in students' bibliographiesassessing changes in students' bibliographies

Outcome of a specific training or series of Outcome of a specific training or series of trainingstrainings

Assessing impact

Bielefeld2006

Bielefeld2006

Importance of the local library for research

2 2 Estimated percentage of the information neededEstimated percentage of the information needed for a publication / paper / report found via the local libraryfor a publication / paper / report found via the local library

• questionnaire/interview after publicationquestionnaire/interview after publication

11 Percentage of material citedPercentage of material cited in academic publications in academic publications that was (could have been) retrieved via the local library.that was (could have been) retrieved via the local library.Citations inCitations in• dissertationsdissertations• student papersstudent papers• publications of a facultypublications of a faculty

Assessing impact

3

Bielefeld2006

Bielefeld2006

Impact on social inclusion and community life

Does the existence of a specific library / of libraries further• individual well-being• the quality of life in a society

- democracy - social inclusion - cultural life - local identity - life-long learning

Method: Assessing public opinion (users and / or non-users) as to the benefits of libraries and special library services

• to the population (indirect benefit)to the population (indirect benefit)• to the individual user (direct benefit)to the individual user (direct benefit)

Assessing impact

4

Bielefeld2006

Bielefeld2006

The financial value of libraries

Assessing impact

5

Assessing time costs (replacement value of a client's time)Assessing time costs (replacement value of a client's time)Users invest time and effort in order to use library services. The Users invest time and effort in order to use library services. The value that they - or their institution - place on that use must be at value that they - or their institution - place on that use must be at least as high as their „sacrifice“ of time.least as high as their „sacrifice“ of time.(Costs are calculated by time and average salary)(Costs are calculated by time and average salary)

Using the contingent valuation methodUsing the contingent valuation method

Willingness-to-pay What would users pay for the What would users pay for the maintenance of a service? maintenance of a service?

Willingness-to-accept Which sum would users accept as Which sum would users accept as equivalent for the deletion of a service?equivalent for the deletion of a service?

Bielefeld2006

Bielefeld2006

For every £1 of public funding the British Library receives each year,

£4.4 is generated for the economy

Using the results

accountability

decision making

resource management

improvement of services

promotion of the library's role

Libraries on the agenda

(Claudia Lux)

Bielefeld2006

Bielefeld2006

Outcome projects

eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information serviceshttp://www.evalued.uce.uk/index.htm

IMLS (Institute of Museum and Library Services): outcome-based evaluation of projectshttp://www.imls.gov/index.htm

ARL New Measures Initiative: several projects - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services)http://www.arl.org/stats/newmeas/index.html

Bielefeld2006

Bielefeld2006

Outcome projectsIBEC: Information School of the University of Washington and University of Michigan School of Information: Toolkit for assessing the impact of information in communitieshttp://ibec.ischool.washington.edu/default1024.aspx

SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing listhttp://www.sconul.ac.uk/activities/performance/impact.html

IFLA Section Statistics and Evaluation: working group on outcome/impact; bibliography of literature and projects worldwide http://www.ulb.uni-muenster.de/outcome.html

Bielefeld2006

Bielefeld2006

(Goethe: Faust)(Goethe: Faust)