Report of the APLAC/PTB Training on the revision of ISO 13528

Report of the APLAC/PTB Training on the revision of ISO 13528

“Statistical methods for proficiency testing by interlaboratory comparison”

12-14 November, 2014

Hong Kong, SAR,

China: Royal Garden Hotel

Summary

A Workshop was approved as a co-operation between the Asia Pacific Laboratory Accreditation Co-

operation (APLAC) and the Physikalisch-Technische Bundesanstalt, Germany (PTB) to provide training for

experts for the creation of sustainable programs in proficiency testing (PT) in the Asia Pacific Region. An

essential part of this infrastructure is expertise in statistical methods that are used for determining

performance evaluations resulting from proficiency testing.

Previous Workshops have focused on requirements for proficiency testing according to ISO/IEC

17043:2010 and the International Standard on statistical methods, ISO 13528:2005. However, the

current Standard from 2005 is not consistent with the requirements in ISO/IEC 17043, and it is being

revised, with final approval and publication expected in 2015. This Workshop is to assist APLAC

Accreditation Bodies in their efforts to anticipate the requirements in the revised Standard, and to allow

application immediately, to assist compliance with requirements for proficiency testing.

The activities for each day are discussed briefly below; the agenda of workshop activities is given in

Annex 1; the list of participants is given in Annex 2; comments from participants regarding the

Workshop are presented in Annex 3.

Workshop Trainers and Host

The Principal Trainer was Mr. Daniel Tholen, USA, Convener of ISO TC69/SC6/WG9 responsible for the

revision of ISO 13528, and Past Chairman of the APLAC PT Committee. He was assisted by Dr. Kunchit

Judprasong, Professor of Nutrition at Mahidol University, Bangkok, Thailand. The Workshop was hosted

by Hong Kong Accreditation Service (HKAS).

Summary of the Workshop

The workshop was a combination of lectures, exercises, and discussions. The lectures covered the

requirements and guidelines in the draft document, followed, at intervals, by exercises using the worked

examples in the draft document. Participants were provided with datasets and, where necessary,

spreadsheets in Excel ® format. There was extensive opportunity for discussion during all lectures and

exercises.

Most of the participants had working experience with proficiency testing and with accreditation of

proficiency testing providers, so there was a good basis of understanding of statistical methods, and of

the requirements of ISO/IEC 17043. By working through the exercises participants were able to gain a

better understanding of the statistical techniques, and an awareness that they are fairly simple to apply.

A Welcome Dinner sponsored by the Host HKAS was provided on the first evening, and offered a good

opportunity for social interaction.

First Day Activities (12 November, 2014)

Following the opening welcoming and introductions, the Trainers described the objectives and schedule

for this Workshop. Mr. Tholen gave an introduction to the Standard, including the Committee

responsible for ISO 13528, ISO Technical Committee 69. The introduction included an overview of the

major changes to the document. The lectures topics are summarized in the schedule, shown in Annex 1.

The exercise for first day was to develop a spreadsheet for implementing Algorithm A, the

recommended technique for calculating robust estimates of participant mean and SD. Dr Kunchit

explained how to perform the calculations in basic Excel functions, and then to apply the Algorithm to

replicating the example in Annex E.1 of draft FDIS ISO 13528. The concluding time for lecture and

discussion covered other parts of the guidelines on statistical design. This closed the first day.

Second Day Activities (13 November, 2014)

The second day continued progression through the draft Standard, including assessment of

homogeneity and stability, visual assessment, and blunder removal. There was a group exercise on

evaluating homogeneity and stability, using the example in Annex E, and use of the kernel density plot.

The afternoon session covered techniques for determining the assigned value and a review of

alternative robust techniques. This included evaluation of the uncertainty of the assigned value. The

afternoon session exercise included calculation with the simple robust techniques (Annex C.2).

Third Day Activities (14 November, 2014)

The third day continued progression through the guidelines and requirements, starting with procedures

to calculate criteria for performance and performance statistics. The exercise covered calculation of

the performance statistics. The afternoon session covered guidelines for qualitative techniques and

graphical methods. The exercise was on basic graphical tools available in Excel, as used in the draft

Standard and in APLAC PT studies.

The workshop closed with final questions and discussion, and an evaluation of the Workshop.

Recommendations

No resolutions were proposed and no follow-up activities are scheduled. The next important step is final approval and publication of version 2 of ISO 13528, anticipated for June, 2015. There could be

actions in different ILAC regions or accreditation bodies to require application of ISO 13528; this will be determined after publication. However, the Trainers and the Organizers of this Workshop recommend the following: 1. That APLAC resolves to require application of ISO 13528 for all APLAC PT studies. 2. That APLAC resolves to make conformance with ISO 13528 a requirement for proficiency testing providers accredited by signatories to the APLAC Mutual Recognition agreement. 3. That APLAC resolves to promote conformance with ISO 13528 as a basis for the ILAC Mutual Recognition Agreement. 4. That APLAC considers sponsoring further training in ISO 13528, as needed by APLAC member bodies.

ANNEX 1: Schedule of Activities: APLAC-PTB Workshop „Revision of ISO 13528: Statistical

Methods for Proficiency Testing by Interlaboratory Comparison“

DAY ONE

Welcome Address and Introduction

WONG Wang Wah, HKAS, Course Coordinator

08:30-09:00

Welcome

Introduction of presenters and participants

Purpose and expectation of the training course

- Objectives

- Content and format

Session One – FDIS 13528, Introduction, Definitions, Basic Model

Presenter: Dan Tholen

09:00-10:30

Reasons for revision of ISO 13528

Statistical expertise needed (Section 0)

Scope and Normative References (Sections 1 and 2)

Terms and definitions (Section 3)

Coffee Break (10:30-10:45)


10:45-12:00 Basic Requirements (Section 4.1)

Basic Statistical Model (Section 4.2)

Lunch Break (12:00-13:00)

Session Two—Section 5: Guidelines for Statistical Design


13:00-14:15

Basis of the design (5.2)

Considerations for distribution of result (5.3)

Considerations for small numbers of participants (5.4 and D.1)

Considerations for report format (5.5)

Presenter: Dr. Kunchit Judprasong

14:15-15:00

Group exercise, examples of design and basic approaches to performance evaluation

Exercise on Algorithm A (Annex C.3)

Tea Break (15:00-15:15)

Session Three—Performance Evaluation


15:15-16:30 Algorithm A (Annex C.3)

Managing “Less Than” values (Annex E.1)

16:30-17:00 Discussion

End of Day One

DAY TWO

Session Four—Section 6: Guidelines for Initial Review of PT items


08:30-09:15 Homogeneity and Stability (6.1


9:15-10:30 Exercise 2 on Homogeneity and Stability



10:45-12:00

Different measurement methods (6.2)

Blunder removal (6.3)

Visual review of data (6.4)

Outlier techniques (6.6)


Session Five – Procedures to determine the assigned value


13:00-14:30

Procedures to determine the assigned value (7.1-7.8)

Comparison of consensus mean and reference value (7.9)

Alternative Robust methods (6.6, Annex C.2)



14:45- 16:30 - Exercise 3 data review and robust estimators using exercises in ISO 13528


16:30-17:00 Group discussion and questions

End of Day Two

DAY THREE

Session Six – Procedures to determine criteria for performance (including standard deviation for proficiency assessment)

Presenter : Dan Tholen

08:30-09:30

Determination of criteria for evaluation of performance (Sections 8.1-8.7)

Limiting the uncertainty of the assigned value (9.2)

9:30-10:30 Performance statistics (section 9.1, 9.3-9.9)


Session Seven – Exercises on evaluation of performance

Presenter : Dr. Kunchit Judprasong

10:45-12:00

Exercise 4 on the evaluation of performance, using examples from FDIS ISO 13528


Session Eight –Graphical Methods and procedures for qualitative data

Presenter : Dan Tholen

13:00-14:00 Techniques for qualitative results (Section 11)

Presenter: Dr. Kunchit

14:00-14:45 Graphical techniques (Section 10)



15:00-16:00 Exercise on graphical techniques

Session Nine – Wrapping up

Presenters : Dan Tholen and Dr. Kunchit Judprasong

14:00-17:00

Remaining questions or discussion

Workshop evaluation

Wrap-up

Close

The End

Annex 2: List of Participants, Home Organisation, and Economy

Name Organization, Economy

1 ARAI, Dr Takashi IAJapan, Japan

2 ALAM, Mr Mohammed Abbas BAB, Bangladesh

3 BALE, Ms Annah PNGLAS, Papua New Guinea

4 BUNT, Dr Mark PTA , Australia

5 CHAN, Ms Yuk Kit HKAS, SAR Hong Kong

6 DAS, Mr Avijit NABL, India 7 ENEBISH, Ms Tegshjargal CGLM, Mongolia

8 HOSAKA, Mr Morio JAB, Japan

9 HOULA, Mr Brahim GAC, Saudi Arabia

10 JIA, Ms Rujing CNAS, PR China

11 KAO, Ms Pao-Chu TAF, Chinese Taipei

12 LABAN, Mr Ian IANZ, New Zealand

13 VENKATESWARAN, Mr Natarajan NABL, India

14 VORASAPAVIT, Ms Chantarat BLA-DSS, Thailand

15 WONG, Ms Fei Ting Standards Malaysia, Malaysia

16 YOUNGJAROON, Mr Komsun NSC-ONSC, Thailand

Annex 3: Workshop Evaluation Comments

Strenghtening regional quality infrastructure networks in the Asia-Pacific region

Evaluation PTB-APLAC Training Course on Statistical Methods for Proficiency Testing

Hong Kong, China, 12-14 November 2014

1

Very poor

2

Poor

3

Average

4

Good

5

ExcellentNumber of

answers Median # < 'Good'

1. Advance information 1 4 7 4 16 Good 5

2. Organization, logistics 4 7 5 16 Good 4

3. Accommodation 1 7 7 15 Good 1

4. Training room 3 10 3 16 Good 3

5. Time schedule and overall

duration 1 11 4 16 Good 1

6. Relevance of the topics

covered in the workshop 9 7 16 Good 0

9. Has this training met your

needs and expectations? 1 9 6 16 Good 1

10. Has this activity provided

sufficient opportunity for

interaction with the presenters

and other participants? 1 5 10 16 Excellent 1

Summary Measures

During the workshop

Learning experience

Before the workshop

1

Very poor

2

Poor

3

Average

4

Good

5

Excellent

Number of


Session Two—Basic design and

planning 7 8 15 Excellent 0

Session Three—Algorithm A

and Censored values 7 9 16 Excellent 0

Session Four—Initial Review of

PT items and results 12 4 16 Good 0

Session Five - Homogeneity and

stability of samples 9 7 16 Good 0

Session Six – Preliminary review

of data 12 4 16 Good 0

Session Seven – Procedures to

determine the assigned value 1 8 7 16 Good 1

Session Eight – Procedures to

determine criteria for

performance (including

standard deviation for

proficiency assessment) 10 5 15 Good 0

Session Ten – Exercises on

evaluation of performance 8 8 16

Good/

Exc. 0

Session Eleven –Procedures for

qualitative data 10 6 16 Good 0

11. How do you rate the relevance of the individual training sections?

1

Very poor

2

Poor

3

Average

4

Good

5

Excellent

Number of


12. How do you rate the general

benefit of the workshop for

your work and the performance

of your home institution? 8 8 16

Good/

Exc. 0

All evaluation items: 0 1 16 166 118 301 Good 17

After the workshop

Comments:

1. Advance information

communication regarding transport to the hotel should be more clear

it's better to get training material one week before (3)

The organizer can arrange a tour for sight visit

a little cold (2)

require more space on the tables for laptops and booklets

Air condition a bit noisy, projector image not always clear

learnt a lot even though not accrediting PT as an AB

2. Organisation, logistics

3. Accommodation

4. Training room

5. Time schedule and overall duration

6. Relevance of the topics covered in the workshop

calculation of z scores by using graphical method

criteria for evaluation

selection for assigned value and planning

calculation by using Excel

Annex D coverage

homogeinity & stability of sample

session 8

exchanging information on ISO 13528 (2)

performance evaluation and qualitative PT

assigned value & uncertainty

session 10

practical session for calculating individual z score/ Group work

more example required for session seven

algorithm 'S', but OK as not there in schedule/agenda

session 5

sharing knowledge with presenters is good, but need to include more groupworks

learnt a lot from AB accrediting PTP and the PT providers

Excellent (PTA, AU)

I can evaluate the result of PT presented by laboratories in the point of statistical method

9. Has this training met your needs and expectations?

11. How do you rate the relevance of the individual training sections?

12. How do you rate the general benefit of the workshop for your work and the performance of your home institution?

10. Has this activity provided sufficient opportunity for interaction with the presenters

7. What was the most important topic for you?

8. Which topic was missing?

by sharing the Excel sheet work and sharing knowledge with my colleagues and other interested parties (2)

briefing session will be held in my organization after attended training

training materials will be distributed to other relevant officers

training to assessors (3), staff AB, committee

come up with a plan on accrediting PTP in future and implementing what was learnt

report to General Manager

reports (2)

Saudi Arabia: trainings, evaluation of PTPs, drafting of PT priorities, participation in APLAC PT committee

TAF Taiwan: hold a discussion seminar for TAF accredited PTP to find out what's the week point of PTP (i.e. what are the statistics we need most)

by conducting further training programs

calculation of PT using graphical/ statistical techniques practically

statistic in RMP

ISO/DEC 17043 (2) and ISO 13528 Planning, Homogeinity, Stability

robust techniques by using Microsoft Excel

design of PT scheme

more core study in session 5, 6, 7

session 10

procedure to determinate the criteria of performance assessment & qualitative PT

ISO Guide 35

organizing more training on these topics and course duration should increase

training related to RMP activities

(Refresher) Training on ISO/DEC 17043 and ISO 13528 (2)

want to share my experience and knowledge (25 years) with PTB and APLAC (Mr D., NABL India)

In-house training in PTPs for selected members of APLAC PT experts (2, Saudi Arabia and CNAS China))

visits/ study tours to organizations/ institutions to have first-hand information/ knowledge

15. Apart from prospective trainings, what activities would you like to have

as part of the PTB support to APLAC (or PAC)?

13. In what way will you disseminate the training outcomes in your home institution?

14. Which topics would you like to be addressed in prospective trainings?

training duration should increase.

Very good training, especially excercise session

please inform us about the courses earlier

TAF Taiwan: Thank you for providing the opportunity to learn what we can't learn anywhere!

suggestion to include print screen shots for each of the spreadsheet calculations

wish to receive the documents before training for preparing

Any further suggestions/ remarks?

Annex 4: Activities of the workshop

Group photo Lecture

Exercise Exercise

Documents

Report of the APLAC/PTB Training on the revision of ISO 13528