PRACTITIONERS REPORT

Application of indicators and quality index as a tool for criticalanalysis and continuous improvement of laboratories accreditedagainst ISO/IEC 17025

Raquel Helena Catini1 Fernando Jose Pires de Souza1

Maria de Fatima Martins Pinhel1 Andre de Oliveira Mendonca1

Vitor Hugo Polisel Pacces2 Igor Renato Bertoni Olivares2

Received: 8 January 2015 / Accepted: 26 May 2015

Springer-Verlag Berlin Heidelberg 2015

Abstract In this work, a model for developing indicators

and quality index was developed and applied, meaning to

evaluate laboratories which have implemented the quality

management system in accordance with the ISO/IEC

17025 requirements. Firstly, the chosen analysis tool (such

as brainstorming and PDCA) to create a development of

indicator and quality index model is discussed. This model

was applied at the National Agricultural Research Labo-

ratory of Sao PauloLaboratorio Nacional Agropecuario

do estado de Sao Paulo (Lanagro-SP). As a result, a set of

indicators was developed and applied which were grouped

to obtain an indicator of unique quality for ten different

laboratory units evaluated. This proved to be an essential

tool for critical analysis and continual improvement of the

quality management system in laboratories exercising the

ISO/IEC 17025.

Keywords Indicator Index Quality Continuous improvement ISO/IEC 17025

Introduction

The implementation of quality management systems for

laboratories in accordance with the ISO/IEC 17025 require-

ments [1] is a necessity in many areas, such as forensic

laboratories [2], chemical pollution in surface water moni-

toring [3, 4], residues and contaminants in food monitoring

[5] and sports doping control [6]. It is actually evident that

ISO/IEC 17025 is established for analytical quality assur-

ance, as it is required internationally, recognized worldwide

and applied by different routine laboratories [7].

In Brazil, the Ministry of Agriculture, Livestock and

Supply (MAPA), published the Normative Instruction

57/2013 [8] establishing new criteria for accreditation,

recognition, scope expansion and monitoring of laborato-

ries, in order to integrate the National Network of

Agricultural Laboratories of the Unified System of Agri-

cultural Health. As the basis for the accreditation process,

this normative instruction depends upon being accredited

by CGCRE/INMETRO (National Institute of Metrology,

Quality and TechnologyBrazilian accrediting entity

which enjoys mutual recognition agreements with the full

members of ILACInternational Laboratory Accreditation

Cooperation) in conformity with the ISO/IEC 17025:2005

[1] standards. Thus, MAPAs National Agricultural

Research Laboratory of Sao Paulo (Lanagro-SP) is

accredited since 2009.

The ISO/IEC 17025:2005 requirements embody all the

ISO 9001 requirements pertinent to laboratories [1],

thereby embodying the process approach [9]. In every

process, it is also possible to apply the continuous

improvement cycle method (PDCA) [10] developed by

Shewhart and refined by Deming. The PDCA cycle is

designed to plan something, defining its objectives and

processes in accordance with predetermined requirements

Electronic supplementary material The online version of thisarticle (doi:10.1007/s00769-015-1143-2) contains supplementarymaterial, which is available to authorized users.

& Igor Renato Bertoni Olivaresigorolivares@iqsc.usp.br

1 National Laboratory of Agriculture, Livestock and Food

Supply (LANAGRO-SP), Raul Ferrari Street,

PO Box: 5538, Campinas, Sao Paulo 13100-105, Brazil

2 University of Sao Paulo Chemistry Institute (IQSC-USP),

Trabalhador Saocarlense Avenue, 400,

PO Box: 780, Sao Carlos, Sao Paulo 13560-970, Brazil

123

Accred Qual Assur

DOI 10.1007/s00769-015-1143-2

(PPlan), execute the process (DDo), verify through

monitoring and measurement of processes and products to

predetermined requirementshaving these results dis-

closed(CCheck), and finally, take actions to correct

and improve the processes involved (AAct).

Seeing that ISO/IEC 17025:2005 is structured according

to the PDCA model, it is possible to assess that it is nec-

essary to pursuit continuous improvement after its

implantation, understanding that among its various stan-

dard items, Requirement 4.15 (management reviews) is a

key aspect. This requirement seeks to widely evaluate

different system information, such as audit results, profi-

ciency tests and corrective actions, among others. This

information must be properly evaluated so that appropriate

actions are taken to evolve (improve) the system. Along

with the many different ways to evaluate those require-

ments, the application of indicators (a tool to obtain

information about a particular reality) and quality index

(aiming at describing more complex phenomena) can be

considered a powerful mechanism, indispensable to eval-

uate a laboratorys performance in its various processes

[11].

Highlighting the importance of ISO/IEC 17025:2005 as

a quality management system for laboratories and the need

for appropriate processes of critical analysis, the objective

of this article is development and application of a model on

the basis of quality indicators for laboratories, with the

intention of contributing to this purpose.

Approach

To develop this work, at the outset, a process for devel-

opment of indicators and quality index was elaborated and

applied at Lanagro/SP to assess ten different laboratory

units.

Development of indicators and quality index model

Choosing the type of indicator, whichever it may be, was

made based on the particular needs of the laboratory.

Brainstorming sessions were held with the departments

leaders to define the indicators, their evaluation parameters

and, subsequently, their assigned degree of importance.

According to Juran [12], brainstorming is a group tech-

nique to generate constructive and creative ideas from all

participants, following these rules:

Good ideas are not supported immediately, as focusmust be on developing new ideas rather than judging

them;

Ideas must be creative opposite to conventional;

Participants are instructed to provide a large volume ofideas in a short period of time;

Participants must follow along after different ideas,expanding, modifying or producing others by

association.

Following the brainstorm sessions, the indicators were

defined and decided how their effect could be rated; they

were grouped to form a single number which would be the

quality index, as developed by Santos [13] for environ-

mental laboratories.

In Fig. 1, a system is shown how to develop the indi-

cators and quality index, structured within the PDCA

concept. Details of each step are given below:

1. Define group responsible for developing indicators

In this stage, a person or a group of people (coordi-

nators) was selected. We found it paramount that this

person or group grasped good people skills and

knowledge both on quality and conflict management.

2. Literature review of relevant indicators

Potential indicators were evaluated, such as Balanced

Scorecard [14] (which considers customers innova-

tion and financial perspective); Harrison and Mengs

proposal [14] (considering the cost of total quality);

Takashina and Floress proposal [14] who use the

National Quality Award model as reference (consid-

ering different criteria such as strategic plans and

leadership, people, processes, customers, the society,

outcomes, etc.); or still, Bezerra and collaborators

proposal [11] (who consider the ISO/IEC 17025:2005

requirements as criteria to develop indicators, such as

the amount of complaints, average time of deploy-

ment of corrective action, number of preventive

actions, proficiency testing results and total number

of errors in records).

3. Laboratory quality management system study

The quality management system was evaluated,

being it already to the knowledge of the group

coordinator, which gave prominence to management

review that covers the systems key points.

4. Selection of experts

Another group of people (experts) was chosen, who

contributed and/or were interested in the indicator

and index results, such as unit heads, supervisors,

internal quality auditors and analysts.

5. Planning brainstorming sessions 1

The coordinators selected potential indicators, which

had already been elected by the specialists during

brainstorming session 1. Also, an initial proposal and

presentation with the works objectives was prepared

to explain each indicator and the reason they were

chosen.

Accred Qual Assur

123

6. Brainstorming session 1 for indicator selection

The brainstorming session was begun by introducing

all indicators, previously selected by the coordinators

and specialists, in an attempt to balance the level of

knowledge. The session was concluded with the

election. During the election (which was performed

using a table to sum each vote for each indicator), it

was proved important to emphasize the need of

finding an optimal number able to adequately repre-

sent the reality of laboratories, minding not to have

an excessive number which would hinder the analysis

and processes.

7. Planning brainstorming session 2

Considering the selected indicators, the coordinators

then assigned a weight of significance to each

indicator, based on the importance of that indicator.

The table was organized and discussed with the

group of experts during brainstorming session 2

(Table 1).

8. Brainstorming session 2 to assign weights to indica-

tors

To conclude the table, brainstorming session 2 was

held with the group of coordinators and specialists to

debate Table 1, making it possible to insert new

criteria for each indicator or modify the weights

assigned to them. When no general consensus was

reached, a voting session was held. To form the index

(which corresponds to the sum of the scores of each

indicator), the total weight summation of the indica-

tors never exceeded 100.

9. Application of indicators and index shaping

Laboratories were assessed through the indicators,

assigning them a final score, so the index was formed

(corresponding to the sum of the scores of each

indicator). A table containing the indicators, criteria

and weights was used. See example in Table 1.

10. Critical analysis of results

The indicators and index obtained were periodically

Fig. 1 Proposed steps todevelop indicators and quality

index

Accred Qual Assur

123

reviewed through meetings with the experts when it

was assessed whether changing indicators or their

weights, selecting other experts, or further adjust-

ments had to be made.

11. Improvement actions

Based on the critical analysis results, possible

changes in the development of indicators and index

formation were proposed and assessed.

Thus, the process of developing indicators and quality

index was structured within the PDCA concept. Figure 1

features the planning stage (Plan), containing steps 18,

including, within the steps, brainstorming sessions. After

the planning stage was completed, data collection for the

development of indicators and index was performed in step

9 (Do). Aiming to check whether the indicators and index

were adequately portraying the reality of laboratories, step

10 allowed holding periodic meetings with the experts

responsible for supporting the development of indicator

process (Check). The indicators, their weights or any other

factor were allowed to be modified based on the results of

the meetings to adjust to the laboratorys current reality

(Act), continuously improving indicators and quality index.

It is important to stress that getting employees to

participate during the selection of indicators was an

essential factor as they were more committed when

implementing them.

Application of the model to develop indicators

and quality index

A model based on the development of indicators and

quality index already presented was applied in 10 Lanagro/

SP laboratory units. At first, a series of meetings was held

with the laboratories coordinators from the units evalu-

ated. Most of these meetings were in the form of

brainstorming sessions to define a criterion to select and

score the indicators.

Ultimately, it was possible to develop 14 quality indi-

cators based on the ISO/IEC 17025 requirements, as shown

in Table 2.

Considering the indicators developed, it was important

to detach the use of indicators linked to the number of

nonconformities, because it was necessary to take into

account the sampling nature of the assessment (i.e., the

more assessments carried, the more findings, and it might

not be a deterioration of laboratory performance). This

way, if the internal or external audits program changed,

the situation needed to be evaluated because it could

influence the comparison of these indicators during dif-

ferent periods.

To evaluate the indicators, a descriptive table with the

scoring criteria was developed (available as Electronic

Supplementary Material). This table was then given to the

laboratories coordinators who were asked to fill them in

(review of the first partself evaluation). In some cases,

Table 1 Example of indicator assessment criteria

Assessment parametera Resultsb Weightc Scored

1. Assessment of inter-

laboratory performance

(last 6 months)

Satisfactory in

every

occasion

5

Warning in some

occasion

2

Unsatisfactory in

some occasion

0

2. No. of non-conformities

in external/

internal audits

05 5

612 2

\12 03. Corrective measures

within deadline

All 5

More than 70 % 2

\70 % 04. No. of improvements

implemented

More than 3 5

13 2

None 0

5. No. of complaints

received

None 5

12 2

More than 2 0

a Enter all selected indicators in Assessment parameterb Add all assessment criteria for each indicator in Results. Des-

ignate a score of importance for each criterionc Assign a weight of significance to each criteriond When development of indicator data collection is done, the result

must be scored in this column, based on the weight of each criterion

Table 2 Developed indicators

Indicators

1. No. of quality management system document (referred to

documents revised within deadline)

2. Implementation of corrective actions

3. No. of internal training

4. No. of nonconformities identified during Audits (Internal and

External)

5. No. of internal nonconformities identified routinely by the analysts

6. No. of preventive actions

7. Quality assurance tools

8. No. of external training

9. Analyzed samples

10. No. of internal audits

11. No. of improvements

12. No. of scientific production and knowledge transfer

13. No. of accredited scopes

14. No. of relevant complaints

Accred Qual Assur

123

the indicator was not applicable at the period evaluated;

therefore, a revised weight column was inserted in the

table, where 0 was added for those cases. This was an

important procedure for the total sum which constitutes the

index. Therefore, the index was calculated by adding up all

figures in the revised weight column, dividing it by the

sum obtained in the score column and multiplying this

ratio by 100 to obtain scores on a scale between 0 and 100.

In this same table, an automatic graphic speedometer

was inserted, capable of graphically displaying how well a

unit performed in each indicator.

After each laboratory units self-assessment (by filling

in the table), it was possible to designate a score for each

indicator and quality index. Figure 2 shows the index for

the different laboratory units (represented by laboratory

unit 1laboratory unit 10) for 2 years (collecting

information to evaluate the indicators was performed for

each period of 6 months, so Fig. 2 shows four different

scores representing each laboratory unit index variation in

a period of 2 years). Figure 3 shows a compilation of the

indicators assessed in all units in a way to check which

indicators are better served by the units.

Conclusions

The application of indicators and quality index stands out

as a powerful tool for continuous improvement. It can also

contribute to requirement 4.15 of ISO/IEC 17025 (man-

agement review). Besides its importance in measuring the

quality management system performance, giving employee

feedback must also be emphasized. The management staff

can disclose the indicators and index onto a full view notice

board, for example. The feedback is intended to engage and

direct employees to continuously improve their exertion,

striving for quality management system excellence.

Regarding the evaluation of the laboratory units, the results

were also considered very satisfactory. Each unit can

occasionally check their indicators in a unique way, based

on the speedometer (as showed in Electronic Supple-

mentary Material) and also verify, in a broader way, its

evolution opposite the ISO/IEC 17025 implementation and

compliance, when comparing its index in different periods

(Fig. 2). It is also possible to assess, in a global way (and as

an excellent laboratory critical analysis tool) aside from

visually assessing all units and their evolution (evaluating

Fig. 2 Temporal development of the quality index of ten laboratory units (numbers on the top of the bars are quality index for each laboratoriesin different periods)

Accred Qual Assur

123

Fig. 2), the percentage of criteria compliance through

indicators (evaluating Fig. 3), making it viable to elaborate

specific training courses, infrastructure, etc., in pursuance

of improving such indicators, thus continuously evolve and

develop the quality management system.

Acknowledgments The authors acknowledge the CNPQ for ascholarship support programme; and the National Laboratory of

Agriculture, Livestock and Food Supply by funds and facilities.

References

1. ISO/IEC 17025 (2005) General requirements for the competence

of testing and calibration laboratories. International Organization

for Standardization (ISO), Geneva

2. Malkoc E, Neuteboom W (2007) Forensic Sci Int 167:121

3. Lepom P, Brown B, Hanke G, Loos R, Quevauviller Ph, Wollgast

J (2009) J Chromatogr A 1216:302

4. Commission European (2009) Commission Directive 2009/90/EC

of 31 July 2009 laying down, pursuant to directive 2000/60/EC of

the European parliament and of the council, technical specifica-

tions for chemical analysis and monitoring of water status. Off J

Eur Union L201:36

5. European Commission, Commission Decision of 12 August 2002

(2002/657/CE) (2002) Implementing council directive 96/23/EC

concerning the performance of analytical methods and the

interpretation of results. Off J Eur Union L221:936

6. Chung KH, Choi GS, Lee W, Cho YH, Lee CW (2006) Accredit

Qual Assur 10:603

7. Olivares IRB, Lopes FA (2012) Essential steps to providing

reliable results using the analytical quality assurance cycle.

Trends Anal Chem 35:109121

8. BRAZIL (2013) Ministry of Agriculture, Livestock and Supply.

Normative instruction no. 57 of December 11

9. ISO 9001:2008 (2008) Quality management systemsrequire-

ments. ISO, Geneva

10. Gupta P (2006) Quality progress magazine, v.39, n.7. Translated

by Maricy de Queiroz Andrade and published in Banas Quality

BQ_172/15 9/2006

11. Bezerra IL, Carreira W, Oliveira RM (2011) Indicators of per-

formance for metrology labs. Bannas Quality Magazine 5:2429

12. Juran JM (1999) Jurans quality handbook, 5th edn. McGraw-

Hill, Maidenherd

13. Santos CR (2009) Laboratory Quality Index (LQI): a proposal for

environmental control laboratories. Doctoral Thesis. University

of Sao PauloSchool of Public Health

14. Martins RA, Neto PLOC (1998) Performance indicators for total

quality management: a proposal of systematization. Manag Prod

Mag 05:298311

Fig. 3 Compilation of indicators assessed in all units (average of indicators compliance)

Accred Qual Assur

123

Application of indicators and quality index as a tool for critical analysis and continuous improvement of laboratories accredited against ISO/IEC 17025AbstractAbstractAbstractAbstractAbstract

AbstractAcknowledgmentsReferences