1

EVALUATION AND QUALITY IN SPANISH ACADEMIC LIBRARIES

Núria BalaguéDeputy Librarian

Barcelona, 14 April 2008

2

Outline:

1. Evaluation and Quality in libraries2. The three levels of quality

assurance1. European level2. National level3. Institutional level

3. Summing-up ideas

1.Evaluation and quality in libraries

4

library resources and planning

library results

user needs and expectations

The quality gap

Library information services value chain

5

= Measurement + Improvement

• Evaluation and quality are complementary terms

• Why to evaluate if we do not intend to improve detected weaknesses ?

• How do we know that we are improving in the right direction if we do not submit to periodic evaluations to confirm it and to help us to define new lines of improvement?

6

Quality Approaches in libraries

• 1960-1990 Statistics, indicators and standards

• 1989-2000 Evaluation and TQM • 1990 – EFQM, ISO 9001, LibQual +

Inputs -> Outputs -> OutcomesResources -> Services -> Users ->

Continuous improvement

7

About Quality • what customers perceive as a result

of the comparison of products or services with others, and through reflecting on their own expectations.

• is dynamic; it is a social construct, always relative and continuously evolving: what satisfies the customer today, will fail to do so tomorrow.

• educational sector has become a consumer item governed by the laws of the market and by the impulses and trends influencing citizens’ perception

8

• Libraries are in transition. Defining and measuring what they do, how much they do, and how well they do it is difficult, because what they do is constantly changing.

(Denise Troll Covey, 2002)

9

A quality-assurance system, a good investment• set of structured processes that serve to plan, direct,

communicate, control and improve the quality within an organization.

• process approach helps to gather, organize and distribute suitable information to suitable people, in optimum conditions of costs and time, with the purpose of being able to make the most suitable decisions.

• develop aspects such as quality policies and quality objectives, establish the functions and responsibilities of staff members, define products and services, identify working processes and the necessary resources to carry them out, and fix mechanisms of supervision, control, prevention, correction and continuous improvement

10

Developing and implementing a quality management system consists of several steps including the following:

• a) determining the needs and expectations of customers and other interested parties.

• b) establishing the quality policy and quality objectives of the organization.

• c) determining the processes and responsibilities necessary to attain the quality objectives.

Quality management systems approach(1)

11

Quality management systems approach(2)

• d) determining and providing the resources necessary to attain the quality objectives.

• e) establishing methods to measure the effectiveness and efficiency of each process.

• f) applying these measures to determine the effectiveness and efficiency of each process.

• g) determining means of preventing nonconformities and eliminating their causes.

• h) establishing and applying a process for continual improvement of the quality management system.

2.The three levels of quality assurance

1. European level

13

EUROPE

• Bologna Declaration (June 1999) European Higher Education Area (EHEA).

• Bergen Conference (May 2005) Standards and Guidelines for Quality Assurance in the European Higher Education Area

proposed by the European Network for Quality Assurance in Higher Education (ENQA).

14

Guidelines for Quality Assurance in the EHEA, Section 1.5

“In addition to their teachers, students rely on a range of resources to assist their learning. These vary from physical resources such as libraries or computing facilities to human support in the form of tutors, counsellors, and other advisers. Learning resources and other support mechanisms should be readily accessible to students, designed with their needs in mind and responsive to feedback from those who use the services provided. Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students.”

2.The three levels of quality assurance

2. National level

16

STRATEGIC LINE No. 3: REBIUN AND QUALITY To enhance, promote and

share resources and expertise that favor potential development of the people who work in University libraries and the quality improvement of management and services

Strategic goals 1st Strategic Objective: Evaluation and Best Practices • To promote, disseminate, lead and

share evaluation processes for the improvement and assurance of quality of library services, the effectiveness and efficiency of the allocated resources and accountability to the institution.

2nd Strategic Objective:Leadership, management and

skills • To continue joint actions to improve

and increase the professionalism of University libraries workers to cope with new challenges

17

1994--

18

19

Spanish National Plans for Evaluation of the Quality of Universities1st Plan (1996-2000)• To promote the institutional evaluation.• To compile a homogeneous methodology and common criteria.• To provide society with relevant and objective information on the level of quality of Spanish universities.

2nd. Quality Plan (2001-2006) continued strategic lines of its predecessor. It also sought to promote the participation of the Autonomous Communities by creating autonomic agencies for the quality assessment of higher education.

72 universities 1.450.000 students

• To provide educational authorities with objective information for making the decisions under their power.

20

Spanish University Libraries Evaluations 1998-2006

• [Previously 3 libraries were evaluated between 1996-98 without specific guide for libraries]

• Some 50 Spanish university libraries were subjected to a formal evaluation process (about 75% of all Spanish university libraries).

• 4 different evaluation guides (high coincidences in the key aspects)

– AQU-CBUC, then assumed by the Council of Universities (based on evaluation UK)

– ACUA (based on EFQM) – ANECA (based on EFQM)– AQU-CBUC (new guide, 2006)

• Same main components of the overall process:

– a self-evaluation

– an external evaluation

– a final report

21

Spanish National Agency for Quality Assessment and Accreditation

Its activities include:

•Improving university teaching, research and management activities.

•Fostering the performance of Higher Education Institutions by monitoring objective and transparent processes.

•Providing public administrations with appropriate information for decision-making within the scope of their authority.

•Providing society with information about the achievement of universities’ aims.

22

Certification of the Quality of University Library Services

EVALUATION CRITERIA Ponderation

1.ELEMENTS BASED ON DATA

1.1. Resources: financial, bibliographical, physical space, etc. 10%

1.2. Results: circulation, interlibrary loans, user education use of electronic resources, etc.

15%

1.3. User satisfaction level. 10%

2. ELEMENTS RELATED TO THE PROCESSES

2.1. Level of integration of the library service in the institution 10%

2.2. Level of relation between the library service and its users: library / teaching, library/ research.

5%

2.3. Management, organization and processes. 10%

2.4. Personnel Policy: education, training and promotion. 10%

3. ELEMENTS RELATED TO IMPROVEMENTS DERIVED FROM THE EVALUATION PROCESS

3.1. Criteria related to the development of improvement plans. 20%

3.2. Criteria related to active participation of library service in innovation and cooperation projects.

10%

23

Objectives• Promote evaluation

• Improve library services• Public recognition

Certification of the Quality of University Library Services

Year Library candidates

Certified libraries Grants total amount

2004 16 7 873,000 €

2005 16 12 877,695 €

2006 9 6 889,644 €

Total 25 2,640,339 €

INTERRUPTED PROGRAMME

24

Evaluation of Catalan university libraries

1st. evaluation of Catalan university libraries (1999-2001). Main aims:• to evaluate library processes and the organizations supporting them.• to assess the added value of libraries in terms of how they strengthen student learning and contribute to the betterment of research.• a cross evaluation of Library Services was performed and a final report was prepared.• the Library Services Evaluation Guide (1999) was based on the methodology proposed in “The effective academic library” (UK, 1995).

2nd. evaluation of Catalan university libraries (2006-2007).• new guide: Evaluation Guide of Library Services and their contribution to the quality of learning and research (2006).• a final report of the cross-evaluation will be published in 2008.

• autonomous region of Spain, 7,000,000 inhabitants, own language (Catalan).• 8 public and 5 private universities. • 200,000 students, 15,000 lecturers and 7,000 administrative staff

25

Evaluation criteria• The library within the university

1. 1. The regulatory framework 1.2. Strategic planning 1.3. Towards the transformation of LS 1.4. LS's relations with CBUC.

• Core programmes Related to : 2.1. Teaching and autonomous learning 2.2. Research 2.3. Digital library and the use of IT 2.4. Cooperation and innovation.

• Organization, management and processes 3.1. Organization 3.2. Processes 3.3. Management of participation and communication.

• Quality assurance 4.1. Structure and organization 4.2. Monitoring mechanisms (improvement plans) 4.3. Mechanisms to know user satisfaction.

• Resources 5.1. Staff 5.2. Buildings and equipments 5.3. Collections 5.4. Budget.

• Results Related to: 6.1. Provision of services and volume of activity 6.2. Satisfaction of users and the staff 6.3. Efficiency in the provision of service

26

Evaluation of Catalan university libraries

First evaluation (1999-2001)

• UB• UAB• UPC• UPF• UdG• UdL• URV

Second Evaluation(2006-2007)

• UB• UAB• UPC• UPF• UdG• UdL• URV• UAO• UVic• UOC

•different guide•more evaluation experience•(almost) similar problems with data gathering

•more libraries•more cooperation•more IT• more user -focused

27

Evaluation of Catalan university libraries

• No negative reactions• Improvements are

evident• Training in quality

improvement• We know better our

weaknesses

• Lack of community involvement

• Some improvements are in hands of the university (not libraries)

• Little support for improvement

External assessment is useful in providing advice, new ideas, and some help from outside the organization, and in the validation of the views of internal reviewers. It works on relevant information (evidences not on intuitions).

2.The three levels of quality assurance

3. Initiatives at an Institutional level

29

Best Practices (UAM)

Quality teams, TQM (UCarlos III, UAB,…)

30

Strategic plans (UPC, UNED, La Rioja...)

31

Balanced scorecard (UPF,...)

32

Service charts (UAB, UPF, UPC, …)

33

User surveys (UPC, …)

34

Affect of Service

Empathy

Information Access

Personal Control

Responsiveness

Symbol

Utilitarian space

Assurance

Scope

Timeliness

Convenience

Library as Place

LibraryServiceQuality

Refuge

Reliability

Ease of Navigation

Convenience

Modern Equipment

LibQual+ (UB, Granada,… )

35

EFQM (Cádiz, …)

36

Acquisitions Database

Non ConformingDatabase

Non Documentary Vendors Database

Reading RoomsDatabase

ILLDatabase

VTLS

Corrective and Preventive Actions

Vendors Continuous Evaluation

UAB Libraries ISO 9001 Quality System

Users Surveys, Suggestions and Claims

SERVICE LEVEL AGREEMENT

Internal & External Audits

Annual Review of the Quality System

Quality Indicators

tra cea bi li ty

continuous improvement

37

UAB Libraries quality journey

 90

91

92

93

94

95

96

97

98

99

00

01

02

03

04

05

06

                                   

European projects                                  

Quality groups                                  

ISO 9000 quality system certification                                  

AQU evaluation                                  

ANECAQuality Certification                                  

3. Summing-up ideas

39

ANECA

ISO

CGC-EFQM

Biblioteca universitària i (multi)certificació de qualitat

Multicertification?

40

Quality: Benefits and Opportunities (1)• The adoption of a quality management system is a strategic

decision of each organization• It is not a purpose itself, but a means for the development of

a continuous improvement dynamics within the library.• We may use:

– a model (as EFQM)– a standard (as ISO 9001)– the criteria of some quality prize or quality agency– using only the common sense, unrelated to any “official”

reference.• Different approaches are not incompatible or exclussive, and

can be simultaneous

• Not a restrictive positioning but an integrative approach in a quality management environment opened to different techniques and models

41

Quality: Benefits and Opportunities (2)

• Today’s organizations are being over audited, which indirectly means that large amount of money and time are (and will be) invested on evaluation.

• If libraries establish a solid foundation for a formal quality-management system, it will be much easier "to adapt the output format" to the specific requirements of any given evaluation that they will have to undergo in order to support university strategies.

42

Evaluation: Benefits and Opportunities (1)• To confront the challenges and the demands of the European

Higher Education Area (EHEA).

• To consolidate the culture of quality.

• To promote appropriate ethics and values within the organization.

• To find out if there is an overload of bureaucracy.

• To find out gaps in the quality management system.

• A mean to constantly analyze the processes performances.

• No “no-go” areas.

• Detection of incipient problems.

• Possibility of immediate implantation of preventive actions.

43

Evaluation: Benefits and Opportunities (2)

• Evaluation as a routine exercise (positive sense)

• Work team reinforcement

• Encourage openness

• Self-reflection on individual tasks

• Self-consciousness of the importance of one’s own work

• It develops a problem-solving culture

• To provide information, advice, challenge and support for the decision-making

44

But...• It is impossible to evaluate everything and overcontrol is

as bad as undercontrol.

• Creating a culture of assessment is not simple, it means changing the mind set and daily activity of librarians and support staff.

• Internet and e-technologies impact the evaluation process, in that evaluations need to become cognizant of the new structures and paradigms of the e-library.

“The methods for assessing new resource delivery and the skills required to conduct such assessments evolve at a slower rate” (D.Troll).

• The objective and criteria of quality evaluations should be gradually improved.

45