IR 056 295 AUTHOR Cotta-Schonberg, Michael TITLE · 2014. 5. 14. · DOCUMENT RESUME ED 405.866 IR 056 295 AUTHOR Cotta-Schonberg, Michael TITLE Performance Measurement in the Context

DOCUMENT RESUME

ED 405.866 IR 056 295

AUTHOR Cotta-Schonberg, MichaelTITLE Performance Measurement in the Context of Quality

Management.PUB DATE [95]

NOTE 14p.; Paper presented at the NorthumbriaInternational Conference on Performance Measurementin Libraries and Information Services (1st,Northumberland, England, August 30-September 4,1995).

PUB TYPE Viewpoints (Opinion/Position Papers, Essays, etc.)(120) Reports Evaluative/Feasibility (142)Speeches /Conference Papers (150)

EDRS PRICE MFO1 /PCO1 Plus Postage.DESCRIPTORS Academic Libraries; Case Studies; Cost Effectiveness;

Efficiency; *Evaluation Methods; Foreign Countries;Higher Education; Librarians; *LibraryAdministration; Library Facilities; Library Planning;Library Role; *Library Services; 'MeasurementTechniques; *Organizational Effectiveness;Organizational Objectives; Staff Development;Strategic Planning; Users (Information)

IDENTIFIERS Denmark; *Performance Indicators; *Service Quality

ABSTRACTThis report describes Copenhagen (Denmark) Business

School Library's (CBSL) use of performance measurement to manage thequality of library services and facilities. The CBSL qualitymanagement program consists of: a statement of strategic goals ofdevelopment and operation, including a focus on users, good service,efficient internal functions, and optimal use of resources; a programfor staff training and development with plans for manuals foroperations and service; an evaluation procedure; and a procedure forcorrective action. Performance measurement is a necessary instrumentfor quality management. It makes possible the assessment of whetheror not a library fulfills its goals and objectives, and whether ornot particular objectives remain relevant. Performance measurementcannot provide a totally objective and comprehensive image of alibrary's function, but is still necessary for a quality consciouslibrary. Management is an art, not a science, and decisions must bemade on the basis of approximate knowledge and intuition. The reportcovers: (1) librarians and management theories; (2) performancemeasurement, evaluation, strategy, corrective action, and the CBSLquality management program; (3) the concept, purpose, and types ofperformance measurement; (4) a fictional case study measuringperformance of an interlibrary loan program; and (5) problems ofperformance measurement that affect quality management. A briefdiscussion with the author is provided. (Author/SWC)

***********************************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

***********************************************************************

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

This document has been reproduced asreceived from the person or organizationoriginating it.

Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

Performance Measurement in theContext of Quality Management

by Michael Cotta-Schonberg

BEST COPY AVAILABLE

2

"PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

Marie Wildt

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

KEYNOTE PAPER

Performance Measurement in the Context ofQuality Management

Michael Cotta-SchsrnbergDirector, The Library, Copenhagen Business School

1. Introduction: Librarians andManagement Theories

1.1 THE PROLIFERATION OF MANAGEMENTSYSTEMS

As you are aware, not a month goes by without theappearance of a new management theory, a newmanagement gum, a new management bestseller.Sometimes, the newness consists in demolishingwell-established theories and practices of othergums, like Henry Mintzberg's The rise and fall ofstrategic planning, which arrived at the CopenhagenBusiness School (CBS) just after its governing bodyhad decided to develop a strategic plan. AsMintzberg is one of the management gums held inesteem at the CBS, his book was received withsomewhat mixed feelings.

In later years, I have myself encountered manage-ment by objectives, participatory management, self-governing groups, team-building, consensus man-agement (which I absolutely loathe and consider tobe a violation of the human right of disagreementand of the noble culture of contradiction), strategicplanning, corporate turn-around, the virtual corpora-tion, the slim corporation, management by walkingaround, benchmarking, mentoring, the five-minutemanager, total quality management and ISO qualitycertification. Most recently I have read about 180degrees management and - a month later - 360degrees management. What these last can be about,I do not know, but I am quite sure that at the nextIFLA conference somebody is going to stand up andgive a paper on `Management by degrees in acade-mic libraries'.

We should not wonder at this seeming fecundityof management theory. It is, after all, not only afield for fertile brains, but also a major industry,involving big money.

Like the fashion industry, the management indus-try systematically depends on continuous changeand newness. The gums are in a perpetual state ofcompetition for lucrative consultancies, grandspeaking fees, and next month's books on the best-seller list.

Actually, competition can sometimes turn into theludicrous or the ethically questionable. For exam-

ple, a recent book entitled The discipline of the mar-ket leaders stayed for 15 weeks on the New YorkTimes bestseller, reaching as high as number five. Itwas then discovered that the authors' consultancyfirm had by various means bought between 40,000and 50,000 copies of the book from bookshops allover the US, particularly bookshops thought to bemonitored by the New York Times (cf. TheEconomist, August 5th 1995, p. 53). So, bewareof bestsellers.

1.2 LIBRARY MANAGERS' FASCINATION WITHMANAGEMENT SYSTEMS

Instead of wondering at the proliferation of manage-ment theories one might more reasonably wonder atlibrary managers' fascination with them, particular-ly in the English-speaking world.

Whenever a new system is presented, it does nottake many months before articles begin to appear inlibrary periodicals on their application in libraries.

There is a positive side to this fascination: it rep-resents an honest recognition that many, maybemost of us, are not born managers, and that manage-ment skills can actually to a considerable extent belearnt or developed. Also, the use of one system as aframe of reference can help to develop one's ownconsciousness about management issues in generaland in particular.

On the negative side, the fascination with man-agement theory is probably also due to feelings ofinsecurity and inadequacy, to some extent but notexclusively caused by the major technologicalupheavals in the library world.

Such fascination may lead to blind acceptance ofone system, uncritical dependence on the guru, cog-nitive blindness towards divergent stimuli from theenvironment and self-imposed limitation on avail-able options for action.

1.3 'BHIKKUS, YOU MUST BE YOUR OWNSALVATION'

There is no saviour system out there, waiting to bediscovered and able to solve one's managementproblem. Learn from the hinayana Buddha's mes-sage to his monks: Bhikkus, you must be your ownsalvation!

In conclusion, a library manager should carefully

51

3

Michael Cotta-Schonberg: PM in the Context of C.321)0111), Management

and critically select all relevant managementinsights, apply them to her own library and create aflexible management system adapted to local andpersonal circumstances.

This is what we have been trying to do at theCopenhagen Business School Library (CBSL).

2. From Problem Solving to QualityManagement

phe development of performance measurement1 and quality management at the CBSL will be

related here. Through this process various importantelements of quality management have been identi-fied and related to each other in a way which maybe of interest to other libraries pondering whetherand how to proceed in this area.

At the outset, it has to be admitted that this devel-opment as a whole was not the result of a majorplanning exercise based on an overall view of thingsand a deep knowledge of management theory. Theperspective, though, was clear: how things look tothe user. It turned out that one phase of developmentnaturally led to another, and in the end we havecome up with a fairly coherent 'system', eventhough we somehow did it all backwards.

2.1 WHAT ARE WE DOING? PERFORMANCEMEASUREMENT

In 1988/89 the CBSL was confronted with a numberof serious problems: other libraries in the area wereconstantly complaining about being swamped byCBS-students (about 15,000) seeking service theyseemingly could not get in their own library; thebudget was grossly inadequate in terms of the tasksdemanded of the library; the major transition tocomputerisation was becoming urgent; there was afeeling that internal processes were not efficient.

In this situation it was decided that the first prior-ity would be to get to know what we actually weredoing.

We already had a set of basic statistics, ie. thosereported on a yearly basis to the Danish library sta-tistics published by the National Librarian's Office.These statistics comprised essential figures on col-lections, loans and resources.

To these statistics were added a series of data onoutput, eg. fill rate and delivery times for inter-library loans from other libraries, and collection use.Special emphasis was laid on data concerning theavailability of books.

To the output data were added data on internalprocessing functions, a major one being the time ittakes for new books to reach the shelves after being

52

received in the acquisitions department. This rathersimple measure actually reflects the overall func-tioning of a number of operations, acquisition, clas-sification and indexing, descriptive cataloguing,physical preparation and registration in the lendingmodule of the computer system.

The development of the performance measure-ment system is an ongoing process. It received newimpetus at later stages with the addition of data onefficiency and effectiveness of operations and ser-vice as well as on the composition of the user group.Every year some measures are cancelled, new mea-sures established and others revised even though themodifications are now comparatively few from yearto year.

Next year we shall be looking at the performancemeasures proposed by ISO and IFLA and take overwhat seems useful for us. Also a serious attempt atincluding cost data has to be made.

Our performance measurement system probablynever will become completely finished since newdevelopments in service and working procedureswill have to be monitored, but a stable form isexpected to be established in two to three years.

2.2 ARE WE DOING IT WELL? EVALUATION

Quite soon in the work with performance measures,it became apparent that they did give us a much bet-ter picture of what we were doing as a library butthat used properly they were limited to the descrip-tive sphere. They did not in themselves tell us if wewere doing a good job.

However, to management it was important to beable to pass a value judgment on the work of thelibrary, to be able to conclude whether or not wewere doing a good job, and if this or that particularactivity was satisfactory or not.

In connection with a set of fortuitous external cir-cumstances, among them the interest and support ina project on the part of the Ministry of Education, itwas decided to establish a program of library evalu-ation. The main purpose of the project was to evalu-ate the 'goodness' of the library. The program wasto be ongoing and aim at annual or biannual evalua-tions.

To ensure independent expertise, Maurice Linewas engaged as a consultant, which has led to a veryfruitful and constructive relationship.

The first evaluation exercise was performed in1991/92. It was based on a set of performance mea-sures and - to a limited extent - on qualitative datafrom interviews with users. The third evaluationexercise covering 1994/95 has just ended. As withour performance measurement, the evaluation pro-

Michael Cotta-Schonberg: PM in the Context of Quality Management

cedure is developed continually but will probablyfind a more stable form in two to three years.

2.3 ARE WE DOING WHAT WE WANT TO DO?STRATEGY

Already in connection with the first evaluation exer-cise it became apparent that subjective evaluationbased on the managers' more or less conscious orarticulated perceptions and values are satisfactoryneither in general nor to the managers themselves.For a value judgment to work in an organisationalcontext of our type, clear criteria of value should beformulated and approved by the responsible body, inour case the Library Board.

Three types of such criteria can be distinguished:Firstly, there are norms and standards approved bycompetent external authorities and associations.Examples are charters for public library servicesissued by local authorities in England, and guide-lines and norms for library services from Americanlibrary associations. Secondly, best practice may beused, obtained by comparison with other libraries(benchmarking) or by comparison of one's ownperformances at different times, mostly from year toyear. Thirdly, many libraries now have a set ofclearly formulated goals in the form of strategies,plans or sometimes service level agreements, oftennegotiated with and approved by the library's acade-mic mother institution.

At the CBSL, we chose the last option in the formof a strategic plan. We had earlier had to formulatesome goals in connection with long-range planninginitiatives for the whole Business School. Thesewere developed into a fully-fledged plan coveringthe whole spectrum of library activities. The plan isrevised annually and approved by the Library Boardas well as by the Academic Council, the highestgoverning board of the Business School.

Most often strategy is considered to be an expres-sion of major and long-range lines of change anddevelopment. We feel, however, that for a serviceinstitution of our type daily operations and serviceare a major strategic issue. The plan therefore hastwo dimensions. One dimension is developmentboth in terms of the long-range transformation ofthe library from a book-place to an informationnexus and of short-range concrete initiatives ofdevelopment and improvement. Another dimensionis operation, which covers not only service - moreor less embodying a service level agreement - butalso the daily internal functions which support theservice function.

In its present form the Strategic goals of develop-ment and operation of the CBSL comprise a short

53

Mission statement and four major Goals, beingFocus on users, Good service, Efficient internalfunctions and Optimal use of resources. The goalsare made concrete in 60 objectives, each with sub-objectives and specific criteria of success when rele-vant.

The strategic plan is now approaching a stableform, though the concrete projects of developmentas well as the specific success criteria will change.

After the development of the strategic plan theyearly evaluation report has been based on this plan.It is structured after the four major Goals and isdivided into 60 sections, one for each objective.Each section contains

- a statement of the objective with sub-objectivesand success criteria if any

- performance measures

- results of any special studies of the activity inquestion during the year

- a summary of initiatives of development,improvement or correction concerning the activ-ity in question during the year

- an evaluative statement concerning both the con-formance between the approved objective andthe actual practice and any problems indicatedby the performance measures or special studies

At this stage of development, ie. in 1994, theBusiness School decided to develop an overall qual-ity program for the whole school, with the assis-tance ofa British specialist, Lee Harvey.

We had, therefore, to look into the quality man-agement literature and movement. We then decidedthat with our strategic plan, performance measuresand evaluation procedure we already had some ofthe basic elements of a quality management pro-gram. However, two elements of such a programmight be considered to be missing or insufficientlydeveloped, one being the creation of conditions forquality and the other being 'corrective' action.

As for creating the conditions, we already had aformalised procedure for ensuring continuing stafftraining and development and some manuals forinternal operations, but these will have to be devel-oped further as an integral part of overall systematicquality management

2.4 WHAT IF WE DON'T DO WHAT WE WANTTO DO? CORRECTIVE ACTION

As for corrective action, we had of course earlierdone something about functions which were evalu-ated negatively, but this procedure, too, did not have


the degree of consistency and formalisation requiredby a quality management system.

In connection with our evaluation report for 1995we have begun to develop this element. We shallprobably have to find another name for it, since`corrective action' - at least in Danish - has connota-tions of something unpleasant done in penal institu-tions!

In case of a negative evaluation of some activity,three types of response are possible.

Firstly, due to problems with measuring perfor-mance - to be described in part 5 of this paper - it isnecessary to consider whether the performance mea-sures are correct or sufficient. If not, it may bedecided to modify the performance measurement inorder to obtain better or more data.

An example: performance measures showed thatwe did not fulfil our success criteria of a minimum95% fill rate of requisitions for interlibrary loansfrom other libraries. The responsible departmentwas convinced that the fill rate was better than indi-cated by the performance measures and a closerlook at the procedure for collecting the data actuallyshowed that there were some problems both ofvalidity and reliability which resulted in inexact fig-ures giving a more negative picture than the actualcase. As it is usually the other way round, we dou-ble-checked the findings and finally concluded thatwe had to modify the procedure and get better per-formance measures before being able to make anevaluation.

Secondly, if the performance measures stand upunder inspection you should consider whether thenegative evaluation may be due to impossible orobsolete goals, objectives or success criteria. In thatcase you should change the goal instead of the prac-ice. An example: we had a success criterion con-sisting in a maximum mean delivery period ofbooks from booksellers of one month. For three suc-cessive years, this criterion was not fulfilled. Ouronly option however seemed to be to shorten thetime by having books from the US delivered byairmail and generally using rush orders much moreextensively than now. However, this option is verycostly so we have decided to retain our presentpractice and to lengthen the delivery time specifiedby the success criterion. Such a course may be dan-gerous, of course; still it is sometimes necessary:goals and objective are not sacrosanct, they tooshould be under continuous scrutiny and revision.

Thirdly, if one is satisfied with the quality ofone's performance measures and wishes to upholdthe existing goals, objectives and success criteriaone will obviously have to initiate remedial mea-

sures. If this is not done, the whole quality processloses its meaning and credibility.

A remedial measure may take the form of rein-forcing the existing procedures. I believe it is thesecond law of thermodynamics which can be restat-ed to mean that things fall apart naturally and bythemselves. There is always a tendency not touphold existing rules, not necessarily because staffhave decided that they are not good. Rules whichare not actively upheld and reinforced will eventual-ly fall into disuse.

On the other hand, if a procedure is not observedit may be because staff at grassroots level have just-ly determined that something is wrong with theprocedure. In that case one should abolish, changeor develop the procedure - in consultation with rele-vant staff, evidently.

If the problem is due to staff lacking the neces-sary qualifications or motivation for implementingsensible procedures, staff development and trainingmeasures are indicated, and if that does not help itmay be necessary to shift staff to other tasks.

2.5 THE QUALITY MANAGEMENT PROGRAM OFTHE CBSL - 1995

As you have heard, we have not only done it all ourway, but we also did it backwards, starting from aproblem and not from a decision to have a qualitymanagement system.

Our quality program is still in a period of devel-opment. It may attain a more stable form in someyears, but it will most probably be in a state of con-tinuous change not only caused by new insightsfrom management theory, but also by changes in thelibrary and information service environment.

As of this year, our quality program contains fourmajor elements:

- a statement of strategic goals of development andoperation consisting of a mission statement, 4major goals, 60 concrete objectives with sub-objectives and specific criteria of success whenrelevant

The objectives under the major goal of Goodservice together form the substance of a servicelevel agreement with the Business School

- a program for staff training and development andfor developing a set of manuals for operationsand service

- an evaluation procedure based on performancemeasurement and other relevant information

- a procedure for corrective action comprisingrevision of performance measures, revision of

64

Michael Cotta - Schonberg: PM in the Context of Quality Management

goals, objectives and success criteria, and reme-dial measures in the form of reinforcing ordeveloping procedures and specific staff trainingand development

Over the last year, we have pondered whether weshould go further and implement a Total qualitymanagement program or seek quality certificationbased on ISO 9000. Right now it is my feeling thatwe should not do this. For two reasons: firstly, Ibelieve that our quality program as it stands todaygives us the essentials of quality management andthat we would not benefit truly by submitting to theformal requirements of TQM and ISO certification.

Secondly, we really cannot expand further theresources spent on this whole process. Some weshould use, definitely, but we must also ensure ade-quate resources for current operations and thevery - important development tasks necessary dur-ing these years.

Thirdly, there is the matter of perspective. Asmanagers we do not want ourselves or our staff tobecome engrossed in procedures of operations andmeasurements, evaluation and in general control.Even more important is it to keep the institutiontuned to the winds of change and actively engagedin the major upheavals affecting the library andinformation world. These issues are of course close-ly related to quality and quality management, butthey are of a different order. Concern with qualityand control must support the even more vital issuesand not deflect our energy from them.

3. Concept, Purpose and Types ofPerformance Measurement

The following remarks are not intended to be of atheoretical nature, but to reflect the experience

of a practitioner and his attempts to develop a usefulconcept of PM.

3.1 THE CONCEPT OF PERFORMANCEMEASUREMENT

3.1.1 Terminological flux

As you are all aware, PM is still in a considerableterminological flux. The important point is thatthere is not any 'correct' terminology.

Quite probably, the national and internationalefforts of developing standards and toolkits for PMin libraries (The ALA guidelines in the US,SCONUL guidelines and The Effective AcademicLibrary in the UK, ISO and IFLA internationally)will eventually lead to some harmonisation of ter-minology.

Total harmony will never prevail, though. Certainterms are untranslatable from or into English, theinternational language of librarianship. For exam-ple, the very term 'performance measurement'translates directly into a Danish term 'praestations-maling' which has somewhat negative connotationsin our culture. Also the distinction between 'effi-ciency' and 'effectiveness' cannot be rendereddirectly into other European languages which haveonly retained one substantive based on the Latinverb efficere and its related nouns. And the distinc-tion between 'measures' and 'indicators' has givenrise to a definitional controversy in Great Britainwhich I do not consider to be important. It doeshave a certain entertainment value, though, and it isone of those extravagancies of the British intellectwhich may assure their neighbours that the Britishare not quite as pragmatic as they would have usbelieve [murmurs and groans from the audience!].

3.1.2 What it is not

In the literature, performance measurement andevaluation are sometimes used interchangeably.This is one practice I consider to be misleading, atthe risk of contradicting my own dictum that there isno 'correct' use. The reason is that I believe it isessential to try as much as possible to keep descrip-tion, of which measurement forms part, distinctfrom value judgment, which is evaluation.

Another thing that it is not is problem solving. Ofcourse, knowledge about the problem and its extent,provided through performance measurement, isessential but the solution to the problem is anotherthing completely and it is chosen in consideration ofpolicy factors and other factors, sometimes quiteunrelated to the problem as such. The performancemeasures may indicate relevant elements of problemsolving; they do not provide the solution.

3.1.3 What it is

By performance I understand library activities in thebroad sense.

Measurement is generally understood to involvesome form of counting. The problem is: what maybe counted? Evidently entities of a physical nature,eg. things like books, persons or users may becounted, and transactions too.

What about users' attitudes, opinions or satisfac-tion? And staff's for that matter or other stakehold-ers', too? These phenomena are of a psychologicalnature, and they cannot be measured directly.Quantified expressions of the phenomena maybe obtained, but what is counted is how many peo-ple tell us this or that about their sentiments and

55

7


experiences and not the sentiments and experiencesthemselves.

Also, the matter of the degree to which usershave certain sentiments (eg. very satisfied, satisfied,dissatisfied, very dissatisfied) cannot be measureddirectly, only how many people use the differentanswer categories. And of course the categories insuch a division of sentiments (very satisfied throughvery dissatisfied) are not equidistant at all.

Some research on users' perceptions of servicedoes not aim at quantifiable data at all, but at exem-plary qualitative description and analysis. Indeed,some would altogether give up the hopeless task ofquantifying anything - as if qualitative research didnot, also, suffer from unsolvable methodologicalproblems in terms of knowledge and generalisation.

This whole matter is naturally an interestingproblem for specialists on methods in the social ser-vices.

On the level of the day-to-day operation oflibraries I think it is reasonable to include data onusers' satisfaction etc. under the concept of mea-sures as long as it is clearly understood and remem-bered that the measures are not direct measures butonly indirect expressions of psychological states.Also that they are based on question and answer cat-egories which users understand and relate to differ-ently. The same applies to categorisation of degree(eg. much, little, often, seldom). Personally, I findthat these problems are fundamentally unsolvable,and that you have to live with them if you want toinclude data on how people experience our servicefunctions and institutions.

In conclusion and for the purpose of this paper, Iuse the term performance measures in libraries asmeaning quantified data on library activities.

3.2 PURPOSE

Quite simply, the purpose of performance measure-ment at the CBSL is: to provide data for the assess-ment of 1) whether or not the library fulfils its goalsand objectives, and 2) the continuing relevance ofthese goals.

3.3 TYPES

There are various typologies of performance mea-sures.

Input / process / output measures: Input measuresrelate to the resources of the library in terms ofmoney, staff, collections etc. Process data concernthe internal functions of the library (eg. processingtime for new books), and output data describe itsservice product (eg. number of books lent).

Traditional library statistics have concentrated on

8

basic input and output measures. The modem devel-opments in performance measurement have includ-ed process data and extended the area of outputdata. As efficient internal process functions are abasic condition for good service, they should not beneglected in a comprehensive program of libraryperformance measurement.

Objective / subjective measures: the subject-object problem is a basic problem of philosophy andpsychology, which I happily leave to the specialists,cf. my comments above.

For our purpose, I consider objective measures asrelating to physical things (eg. number of books)and subjective measures as relating to psychologicalstates (eg. user satisfaction).

I do not consider objective measures to be moretrue than subjective measures. Both are necessary,but even together and even when obtained underoptimal conditions they will not give a completeand 'true' picture. Angels would not be happy, butlibrary managers will be pleased to obtain sensible,however imperfect data with which to plot theircourse in the murky-muddy waters of everydayoperations.

An interesting development at our institution (theBusiness School) has been an experiment withobtaining subjective measures of the same itemsfrom various stakeholder groups. At the CBSL ithas been tried out by submitting questionnaires onvarious aspects of service not only to users but alsoto the library staff. Interesting differences of opinionbetween users and staff have been demonstrated. Inone case, users were happy with the service at theinformation desk, but staff felt there were seriousproblems eg. relating to new media (CD-ROM) andthey were definitely less happy with their service tousers than users seemingly were. Another case wasthe OPAC, where users expressed satisfaction withthe user-friendliness of the system, while staff werenot very impressed by users' OPAC-competence.This difference prompted a special investigation ofa qualitative nature which concluded that users -contrary to users' own perceptions - are not verygood at using the system and that they are, for thisreason, far from profiting optimally from the infor-mation resources which the library puts at their dis-posal.

Non-comparative and comparative: performancemeasures concerning one institution in a single yearare of course interesting in themselves. However,they become much more interesting when they canbe compared with data from other years and fromother institutions.

Comparative data are fraught with their own kind

56

Michael Cotta - Schonberg: PM in the Context of Quality Management

of problems. Due to changes or differences in defin-ition and practice, comparison across time and insti-tutions may be difficult, sometimes extremely so.Still, they should be tried, and problems solved asthey arise or are recognised.

Whether comparisons between institutions' per-formance measures can be considered to be 'bench-marking' is a matter for debate. If one includesqualitative analysis of best practice in the definitionof benchmarking, mere comparison of quantitativedata may not suffice. Such data must be seen intheir relationship to libraries' mission, goals andobjectives and they do not provide answers to whyand how one library has 'better' scores than another.Still, it is interesting to create a horizon of compari-son and the comparison of quantitative data mayalso be quite relevant as first steps in a process ofidentifying, describing and analysing 'best practice'.

Simple and composite: simple measures aredirect measures of something. Composite measuresare obtained by relating simple measures, usually bydivision.

Composite measures (sometimes termed as indi-cators in contradistinction to measures) are neces-sary for a deeper analysis of phenomena and theyare essential to comparison over years and acrossinstitutions. It is interesting to know that a librarylends 300,000 books a year. It is even more interest-ing to know that it lends ten books per FTE student,especially if you want to know how the library per-forms in relation to other libraries. To know that onelibrary lends 400,000 books and another 100,000 isnot very interesting in itself. What you would reallylike to know is how the circulation figure relates tothe budgets of the libraries and the number of stu-dents - and the rate of registered use to unregistereduse in reading rooms. And then you would still needto know the basic acquisition policies of the institu-tions, of course: do they acquire books to cover theurgent needs of students mainly, or do they acquirebooks 'for the centuries and the scholar showing upin 200 years', or do they acquire books because theyare responsible for preserving the national literaryculture?

4. A Case: Interlibrary Loans fromOther Libraries

4.1 INTRODUCTION

In order to illustrate the points and ideas developedabove, I shall now present a case. It is based on alibrary resembling the CBSL, but the data givenbelow are not identical with ours, and they are

9

selected with a view to clarify certain basic notions.Imagine a library being part of a higher educa-

tional institution and serving a primary user groupof 10,000 students (1-Ths) and 600 academic staff.Its yearly budget is 2.5 million pounds and it spends350,000 pounds on acquisitions, including 10,000pounds on ILL from other libraries. Its collectionsamount to 250,000 volumes. It pursues an activeweeding policy and aims at a no-growth library of300,000 volumes in the year 2000.

The activity concerning interlibrary loans fromother libraries has been stagnating for a number ofyears. Formerly, almost all loans were for teachersat the CBS; in the most recent years there has beenan important relative rise in the number of loans forstudents. It is believed that the last development isdue to students becoming familiar with the largebibliographic databases on CD-ROM and in thediminution of the - already - small fee for IL loansfrom abroad. This fee is not imposed in order torecover costs but to regulate behaviour and toensure that only items really needed are requested.

Whereas teachers often do not need very speedydelivery of materials borrowed from other libraries,the students need rapid delivery because the materi-als are to be used in papers to be finished withinweeks or a few months - or yesterday!

The library is still predominantly using traditionalmethods of ILL. Only recently has it begun to useonline and fax-based methods of delivery and sys-tems like UNCOVER are still only used to a verylimited extent.

4.2 OBJECTIVES

The objective of our fictive library concerning inter-library borrowings is the following:

- To provide reliable and speedy loans from otherlibraries and information systems

The success criterion is:

- A fill rate of 95% of all requisitions receivedfrom users and a mean delivery period of 21days

4.3 THE PERFORMANCE MEASURES

The performance measures are given in three tables,one with objective data, one with subjective dataand one with comparative data.

In Table 1 the 'fill rate' is the number of requisi-tions from users fulfilled within standard delays.

`IL loans/all loans %' means loans to users ofmaterials borrowed from other libraries as a per-centage of all loans.

It is seen in this table that:

57


Table 1

Objective performance measures of IL borrowings

1992 1993 1994

For teachers 2,500 2,500 2,500

For students 500 600 700

All 3,000 3,100 3,200

Fill Rate 93% 94% 95%

Delivery Time (mean) / days 25 23 21

IL borrowings / all loans 1.0% 0.9% 0.8%

Cost per loan / D.Kr. 75 77 79

- the number of interlibrary borrowings is increas-ing very slowly, and only due to an increase ofborrowings for students

- The fill rate has risen over the last years to justfulfil the strategic requirement of 95%

- The delivery time has fallen over the last years tojust fulfil the strategic requirement of 21 days

- The interlibrary borrowings activity forms a verysmall and decreasing part of the total lendingactivity of the library

- The cost of interlibrary borrowings is going up alittle, but is stagnating if inflation is accountedfor

Table 2 gives the (very!) fictive satisfaction rates of4 groups: 2 user groups, ie. academic staff and stu-dents, the library staff, and the library management.The indifferent, unsatisfied and very unsatisfiedusers are not given - for simplicity's sake.

It is seen in this table that concerning the fill rate

- teachers are consistently and highly (99%) satis-fied

- students are consistently satisfied at the 80%level

- library staff satisfaction increased over the yearsfrom 50% to total satisfaction coinciding withthe achievement of the strategic objectivelibrary management only became satisfied whenthe strategic objective was fulfilled in 1994

Concerning the times of delivery

- teachers were consistently satisfied at the 90%level

- students' satisfaction level decreased from 80%to 40%

- library staff satisfaction followed the same pat-tern as for the fill rate

- library management satisfaction fell from 50% to0%

1

Table 2

Subjective performance measures of IL borrowings

Satisfied or very satisfied 1992 1993 1994

Teachers

Fill Rate 99% 99% 99%

Delivery Times 90% 90% 90%

Students

Fill Rate 80% 80% 80%


Library Staff

Fill Rate 50% 75% 100%


Library Management

Fill Rate 0% 0% 100%


Table 3

Comparative performance measures of IL borrowings

Lib A Lib B Lib C Lib D

All IL borrowings 3,200 4,500 5,000 5,500

IL borrowings / all loans 0.8% 5.0% 2.0% 4.0%

Cost per loan / D.Kr. 79 120 95 102

In Table 3, library A is our case library, and the oth-ers are libraries of the same type and those whichare most often used by library A for comparison.

It is seen that

- the case library has the smallest number of inter-library borrowings both in absolute numbers andas part of the total lending activity

- it also has the smallest cost per borrowing

4.4 EVALUATION ON THE BASIS OFPERFORMANCE MEASURES

The first question addressed in the evaluation exer-cise is the following:

Is the actualjractice of the library in conformitywith the strategic objective and the success criteri-i2n2

The answer is: yes. It is satisfactory that the libraryhas now come to fulfil its strategic objective con-cerning interlibrary borrowings.

The second question is:

Po all the available data point to a satisfactory situ-ation concerning interlibrary borrowings?

58


The answer is: no. The decreasing satisfaction ofstudents and management with the time of deliveryis not satisfactory, even if the strategic objective isfulfilled.

Fictive follow-up studies of this phenomenon showthat students are simply getting more demanding,particularly as they get more attuned to instantonline contact with an increasing number of infor-mation resources. Management is unhappy becausemajor systems for rapid (24-48 hours) online or faxdelivery of periodical articles have now becomeavailable with the result that delivery times whichwere satisfactory in the earlier period have nowbecome unsatisfactory.

The third question is:

How would you evaluate the activity concerninginterlibrary loans on the basis of these data?

It is satisfactory that the strategic objectives are nowbeing met. However, the success criterion concern-ing the delivery time has been made obsolete by theinformation technological development and shouldbe revised.

The fourth question is:

If you have mentioned any problems in the evalua-tion. how would you_propose to address them?

Three types of corrective action have been men-tioned earlier: improving the data, revising strategicobjectives or their success criteria, undertaking spe-cific initiatives of development.

In our case, all three will be employed.

- A special investigation of how many requisitionsactually concern periodicals available on high-speed document delivery systems will be made.If a sufficient number proves to be available,then

the success criterion for speedy interlibrary bor-rowings will be lowered from 21 to 14 days (asa cautious beginning)

- ILL procedures will be changed so thatUNCOVER and similar systems will be used inpreference and if possible to other, traditionalpartners

The main notion to be illustrated by this case is thatperformance measures should be used both forassessing whether strategic objectives are being metand if strategic objectives remain relevant. Often,subjective performance measures may be the bestindicator of the continued relevance of strategicobjectives.

11

5. Problems of PerformanceMeasurement Affecting QualityManagement

One thing with performance measures isunavoidable: you can never feel safe! Some

definition suddenly turns out to cover aspects youhad not thought about; somebody 'on the floor'changes the way of counting something, the demonin your machine develops acute insanity - as when acomputer at a major library started to count onlyevery second loan - or there has been a misunder-standing between managers and programmers con-cerning some parameter of automatically producedmanagement information.

Before going into various problem types, a pieceof advice: stay calm! Try to control the informationgathering process, start with clear definitions, estab-lish simple procedures, have someone in chargewho has a good general knowledge of your library'sfunctions and a feeling for numbers and who cansense when there must be something wrong with thedata. This advice is so good that I shall eventuallyfollow it myself!

However, whatever you do the error-free systemis impossible. Recognise that and simply correcterrors and misunderstandings as they are discov-ered. Sometimes, you may have to declare certaindata lost and you will lose comparability over theyears. Well, life goes on.

Also, if you are experimenting with new forms ofcomplex data you may reach a point where itbecomes necessary to declare that the problems indefining and collecting them cannot be solved ade-quately, something still stinks. In that case, cut themeasurement. Even if it may be embarrassing inview of your earlier efforts, possibly in the face ofstaff opposition.

5.1 VALIDITY

By validity is meant simply that the measure actual-ly measures what it is supposed to measure.

An example: We have been measuring the time ofdelivery of books from our book-agents, countedfrom the day our order is sent to the day the book isreceived. The time consistently turned out to be(and still is) surprisingly high. However, only after acouple of years we became aware that the CBSLorders quite a large number of books, on the basis ofpre-announcements, which have not been publishedat the time of ordering. Our measurement of deliv-ery time had thus somehow been messed up withdelays of publication. The problem was solved byexcluding books, which we knew had not been pub-

59


lished at the time of ordering, from the measure-ment. We could not solve the problem of bookswhich had actually not been published but wethought they had. Still the measurement today ismore valid than it was before we made the correc-tion for books known to be unpublished.

Another example: the fill rate for requisitions ofbooks through interlibrary loan from other librariesis calculated on the basis of number of requisitionssent to other libraries (B) and the number of booksetc. received (A) - as A/B%. Three problems havebeen discovered. Firstly, it turned out out that if wedid not get the material requested from a library, wewould send the requisition to another library. Wewould then count the requisition twice. As we onlyobtained the material once, the fill rate would be50%. However, there would only be one requestfrom the user, and she actually obtained it! So wehad a problem of validity. We solved it by startingto count only the first requisition to a library, andnot the repeat-requisitions to other libraries.

Secondly, at some point it turned out that users'requests were also counted as ILL requests in thosecases when our library decided to buy the bookrequested. As the request was counted as an ILLrequest by the ILL department, but the book was notreceived by that department (but eventually by theacquisitions department), the book was not countedas a fulfilled ILL request. So the fill rate for thisrequest was 0%, even though the book would actu-ally be obtained for the user. Again a problem ofvalidity. We solved it by starting not to countrequests transferred from ILL to acquisitions. Still,how should we deal with user-requests transferredto the acquisitions department when the book wasnot actually received from the bookseller? At pre-sent we have chosen to ignore the problem, so westill have a validity problem somewhere, but thepure ILL measure is more valid now.

Thirdly, a number of ILL requests are based onfaulty and unverifiable bibliographic informationfrom the users. They concern unexisting materialsor materials which exist by another name. As thelibrary's requisitions cannot be met, these userrequests lower the fill rate to an unknown degree.However, as we now have a fill rate of 98% of ourILL requisitions to other libraries, we have decidedto ignore the problem.

So, you will not be surprised that having discov-ered three problems with this fairly simple measureI cannot help wondering what problems we mightnot yet have discovered in this regard.

12

5.2 RELIABILITY

Is a measurement performed consistently and cor-rectly?

An example: One of the departments of theCBSL was responsible for reporting monthly thenumber of article copies sent to other libraries. Themonthly number was usually about 20. Then, onemonth the number rose to about 200. I noticed it butdid not have it checked, as it might be because ofsome special order from another institution. Whenthe c. 200 figure was repeated during thenext two months I asked the head of the departmentwhat was happening. She answered that otherlibraries were discovering the excellent holdings ofthe department and the promptitude with whichrequests were fulfilled. Not an eye was dry, as wesay in Danish. However, a week after she calledback, very embarrassed, that actually it was a newtechnical assistant who had started to count thenumber of photocopied pages instead of articles.

These things will happen all the time, and someof the problems will reflect the way your librarytrains new staff members, such training being ofcourse an essential element of quality management.

Sometimes the unreliable (or invalid) measuresgive a false-better image of your library's doings,sometimes a false-worse one. At any rate, thegreater the unreliability of your measurements, theless you are able to assess the functions of yourlibrary and whether you are meeting your own goalsand objectives. You can then only hope that you donot run into a major uncharted iceberg-problem andsink with all hands.

5.3 VERBALISATION

For the benefit of your audience - the readers ofyour quality report, staff, university managementetc. - you want to verbalise those complicated fig-ures in the tables.

`This table shows that . . . '

The first problem arising is one of translating fig-ures into words. When is a 25% increase a 'largeincrease', when is it 'small'? When is it 'significant'or 'considerable'? Even if you do take care aboutusing such words consistently and in considerationof the underlying absolute figures or their degree ofcomplexity (eg. the use rate of your collection), youmay easily slip into using language which somehowreflects your personal values. As the measuresshould provide a solid basis for a common under-standing of the situation of the library, this is a prob-lem.

60


5.4 INTERPRETATION

A most common error concerns causality. A certaindevelopment is clearly indicated by some figures,and you triumphantly or dejectedly conclude thatthis or that change of policy or unfortunate externalevent was responsible for the development.

An example: recently the CBSL decided to lowerthe fee for articles printed from our full text busi-ness periodicals CD-ROM system. The change wasfollowed by a 'significant' increase in the numberof copies requested by users. As I had myself pro-posed the lowering of the fee, I easily concludedthat the rise in the number of copies was due to thelowering of the price. My deputy, however, who isresponsible for the library's economy and who hadopposed the measure, countered that the develop-ment was due to the quite predictable increase ofstudents' familiarity with the system. We may bothbe right, of course.

The point is that the measure in itself did not tellus about the reason for the development of the fig-ures.

It would be inhuman to forbid interpretation andexplanations of performance measures, even con-cerning causality. However, care must be shown,and before you publicise your findings you mightprofit from having your interpretations examined byinformed, intelligent people - preferably some whodislike you vehemently. You may suffer in theprocess, but you should end up with more refinedinterpretations.

5.5 RECEPTION

In the total quality process it is important how yourperformance measurement is 'received' by yourstakeholders.

In relation to the CBS, ie. our academic motherinstitution, it has been rather encouraging that theSchool authorities react positively towards thelibrary's efforts in the line of performance measure-ment and quality management. The credibility ofthe library as a service institution is raised, thoughsometimes - embarrassingly - our good intentionsare confused with the actual thing.

A problem is that external parties may easily fix-ate on certain very understandable measures, likethe number of loans. As long as it goes up, fine.What will happen when it goes down because weshall be actively promoting our users' direct con-nection with international online informationproviders, thus effectively teaching them to bypassthe library?

Another problem is unwanted comparisons withother, similar institutions. Why does this library

6 1

13

have fewer loans per student than the library at XUniversity? Well, possibly because we have anabnormally high percentage of part-time students,for example. Actually, I have not met this type ofreaction at our own institution, but I am told that itcan be a real problem in countries, like England,where generalised systems for comparison of per-formance measures may be important for allocationof funds.

At any rate, I do believe that a consistent andcredible program of quality management and perfor-mance measurement will raise the library's standingin the institution of which it forms part.

In relation to library staff the situation is highlycomplex.

Performance management may be resented bystaff as basically a mechanism of control and atime-consuming waste of resources. The problem isaggravated in cases when the performance measuresdirectly concern deficiencies and faults at stafflevel, eg. the number of cataloguing errors.

In this respect libraries are very different, andgeneral advice difficult.

The much desired total mobilisation of staffrequired by the formal TQM systems I consider tobe impossible in cultures with low degrees ofauthoritarianism and absence of the mystique of theleader. I do not consider it to be a coincidence thatthis requirement (of total staff mobilisation) wasformulated in Japan and in the US.

Personally, I distrust management by mystique,by seduction or by charismatic infusion of 'motiva-tion' by leaders into staff.

Taught by experience, I am also sceptical aboutthe possibility or desire of each individual staffmember to relate to the total quality picture andmanagement of the institution.

I do believe that managers should make a reason-able and intelligent effort at having staff understandand participate in performance measurement andquality management.

On the other hand, I feel that the claims made bythe QM gurus concerning staff engagement in thetotal process are exaggerated. Many staff memberscan perfectly well want to do a good job and actual-ly do it without subscribing to or being deeply inter-ested in managers' grand plans. As long as they do agood job, they do not have to say, 'Hail quality'every morning.

And the users? Users are only dimly, if at all,aware of quality programs and performance mea-surement. What they react to is basically: can theyget what they need - when they need it? Blissfullythey ignore our wonderful intentions and are only


concerned with one thing: the actual service prod-uct.

However, we have discovered that students dotend to react positively to user studies, particularlythose which are carried out in a 'personal' atmos-phere - being questioned over a cup of coffee and abiscuit, preferably by a young nice person of theopposite sex! Comments reflect the feeling that 'it isgood that the library cares'.

There may be some problems of methodology inthis and the effect - if it is true - is an unintendedside-effect. However, if the library really does care,it is OK that the students get the message.

6. Conclusionperformance measurement is a necessary instru-r ment for quality management. It makes possiblethe assessment of whether or not a library fulfils itsgoals and objectives, and whether or not particularobjectives remain relevant.

Performance measurement cannot provide a total-ly objective, 'true' and comprehensive image of alibrary's function. A quality conscious library can-not do without it, though, even if it will always beflawed or incomplete.

So, management will go on being an art, not ascience, and decisions will still have to be made onthe basis of approximate knowledge and intuition,not as results of computer calculations anduntouched by human minds. Fortunately!

Discussion

Don Revill, Liverpool John Moores University: Onthe subject of time and its importance: I do notbelieve time is of the essence. Time is somethinglibrarians think they ought to minimise. No otherprofession gives instant answers. I would say to astudent that if it takes 21 days to get an interlibraryloan, they ought to have requested it earlier. It issaid that taking time to deliver imbues it withimportance. It was worth waiting for. Research hasshown that time is not important to academics but itis to the students. They shouldn't need interlibraryloans. Why worry if it takes a long time. It is some-body else's performance that is at fault: the publish-ers, the transport system, the postal service.

Michael Cotta-SchOnberg: I respectfully disagree.There is always a fast way to do things. And stu-dents are important people too. The students' time isimportant to them, so it is important to us. But to

take one instance that I am keenly concerned about:if a student wants a book that we have actuallybought, and if he or she has to wait six monthsbefore getting it because there is a queue for it, isthat not important to the student? Should we not beconcerned about it?

Don Revill: That is a different problem.

Michael Cotta-SchOnberg: OK, but it is a time prob-lem.

Roswitha Poll, Universitats- and LandesbibliothekMiinster: We are working on a national system ofinterlibrary lending now which will be very quick,`Con Subito'. In a questionnaire to find out desireddelivery times, we asked whether 24 or 48 hourswas too quick for document delivery and askedusers about payment for a quality service, say theequivalent of Ll , £2 or £3 per document. We foundthat 50% to 80% were satisfied with two weeksdelivery. 20% to 30% wanted material very urgentlyand would pay for it. Security - being sure of gettingthe document - seemed to be more important thangetting it quickly.

62

14

4

U.S. Department of EducationOffice of Educational Research and Improvement (OERI)

Educational Resources Information Center (ERIC)

REPRODUCTION RELEASE

I. DOCUMENT IDENTIFICATION:

(Specific Document)

0

I

Title: Performance Measurement in the Context of Quality Management.Note:Also available:

Author(s): Michael Cotta-SchoenbergCorporate Source:

http://www.cbs.dk/people/mcs/bibmcsuk.htmlPublication Date:

Key paper from the 1st Northumbria Conference on Performance Measurement)1995

II. REPRODUCTION RELEASE:In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced

in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproducedpaper copy, and electronic./optical media, and sold through the ERIC Document Reproduction Service (EDRS) or other ERIC vendors. Credit isgiven to the source of each document, and, if reproduction release is granted, one of the following notices is affixed to the document.

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following two options and sign atthe bottom of the page.

Check hereFor Level 1 Release:Permitting reproduction inmicrofiche (4" x 6" film) orother ERIC archival media(e.g., electronic or optical)and paper copy.

The sample sticker shown below will be

affixed to all Level 1 documents

PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL

HAS BEEN GRANTED BY

(1°

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

Level 1

The sample sticker shown below will beaffixed to all Level 2 documents

PERMISSION TO REPRODUCE ANDDISSEMINATE THIS

MATERIAL IN OTHER THAN PAPERCOPY HAS BEEN GRANTED BY

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

Level 2

Documents will be processed as indicated provided reproduction quality permits. If permissionto reproduce is granted, but neither box is checked, documents will be processed at Level 1.

El

Check hereFor Level 2 Release:Permitting reproduction inmicrofiche (4" x 6" film) orother ERIC archival media(e.g., electronic or optical),but not in paper copy.

I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminatethis document as indicated above. Reproduction from the ERIC microfiche or electronic/optical media by persons other thanERIC employees and its system contractors requires permission from the copyright holder. Exception is made for non-profitreproduction by libraries and other service agencies to satisfy information needs of educators in response to discrete inquiries.'

Signhere)please

Copeo.DhagenBusinessLibrary

Signature:

0L',6a02kOrganization/Address:

Handeishoiskolens BibliotekRosenems Alla 31

Schad] 1970 Frederiksberg CTit 3815 3686 - Fax 3615 3663

Printed Name/Position/Tide:

Marie Wildt, LibrarianrirePh.ane Tel +45 38 15 36

Fax +45 38 15 36

E-Mail Address:

hbk/[email protected]

6663

1.1010.

96.08.13

(over)

III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE):

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source,please provide the following information regarding the availability of the document. (ERIC will not announce a document unless it ispublicly available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria aresignificantly more stringent for documents that cannot be made available through EDRS.)

Publisher/Distributor:

Address:

Price:

--- IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER:

If the right to grant reproduction release is held by someone other than the addressee, please provide the appropriate name and address:

Name:

Address:

V. WHERE TO SEND THIS FORM:

Send this form to the following ERIC Clearinghouse:P "if

arrP.3.7.ftr,kal;%T;

ITAW(A

lagikAia.4

However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document beingcontributed) to:

ERIC Processing and Reference Facility1301 Piccard Drive, Suite 100

Rockville, Maryland 20850-4305

Telephone: 301-258-5500FAX: 301-948-3695

Toll Free: 800-799-3742e-mail: [email protected]

(Rev. 3/96/96)