Proficiency Test Working Group, ISTA GMO TF · PT03: Qualitative test A classification (irrespective of the type of method used) of each of the 12 samples as either positive or negative

Proficiency Test Working Group, ISTA GMO TF

OutlineOutline

Introduction:Design of the three conducted PTs

Results:Qualitative testsSub-sampling testsQuantitative testsComparison of the results from PT01, PT02 and PT03

Conclusions

International Seed Testing Association

Design of Design of the threethe three ISTA ISTA ProficiencyProficiency TestsTests ((PTsPTs))

Proficiency Test PT01 PT02 PT03

# of laboratories 43 52 50

# of samples per lab 30 10 12negative 12 3 3positive 18 7 9

0.2 % - - 30.7 % - 3 -1.0 % 18 - -1.4 % - 4 -2.0 % - - 34.0 % - - 3

Events T25/Mon810 Mon810 T25/Mon810

Sample size 300 3000 1500

Qualitative X X XSub-sampling - X XQuantitative - X X

From From 52 52 laboratories laboratories in PT02:in PT02:

Submitted Submitted test test results from results from PT02 and PT03PT02 and PT03


qual. & quant. Test

22 (45%)

qual. & sub-sampling

Test7 (15%)

no results8 (17%)

qual. Test11 (23%)

qual. & semi-quant. Test

12 (23%)

no results5 (10%)

qual. Test15 (29%)

qual. ,semi-quant. & quant. Test

4 (8%)

qual. & quant. Test

16 (30%)

From From 48 48 laboratories laboratories in PT03:in PT03:

PT03: Qualitative testPT03: Qualitative test

A classification (irrespective of the type of method used) of each of the 12 samples as either positive or negative must be done.

40 laboratories reported results for the qualitative test:

96.1% of the 491 samples were reported correctly.


# of samples tested

# of samples reported incorrectly

% of samples reported incorrectly

491 19 3.87122 4 3.28369 16 4.34123 14 11.38123 0 0.00123 2 1.63

0.2%2%4%

all samplesnegative samplespositive samples

Comparison Comparison of qualitativeof qualitative testtest resultsresults(PT01, PT02 and PT03)(PT01, PT02 and PT03)

• The genetic purity of the used material was tested in the sameway for the proficiency tests.

• The percentage of correctly reported results is in the same range.

• The rate of false negative and false positive reported results in PT03 are between PT01 and PT02.

• The false negatives in PT03 are mainly due to the samples with a 0.2% GM content.


Proficiency TestPT01 PT02 PT03

False positive 1.74% 4.96% 3.28%False negative 6.46% 3.04% 4.34%Correctly reported 95.4% 96.4% 96.1%

PercentagePercentage of of laboratories which reportedlaboratories which reported......


Proficiency Test

PT01 PT02 PT03

Correct results 70% 85% 81%

False results 30% 15% 19%

False negative 19% 4% 12%

False positive 7% 4% 5%

Both false neg. & pos. 5% 6% 2%

ComparisonComparison of qualitativeof qualitative testtest resultsresults(PT01, PT02 and PT03)(PT01, PT02 and PT03)

The laboratory reports the % of GM seeds, whether the GM level is ≤ or > 1% and the testing plan:• the number of sub-samples• the size of sub-samples (number of seeds) and• the number of positive sub-samples per sample• the decision Rule


PT03: SubPT03: Sub--samplingsampling testtest

89 % 61 % 100 % classified correctly

(PT02: 44% of the 0.7% and 75% of the 1.4% GM content samples)

0.2% 0.2% 0.2% 2% 2% 2% 4% 4% 4% sample-size# of sub-samples

size of one sub-sample

max. pos. sub-samples

7 1500 12 125 857 1235 13 95 847 neg neg 4 400 324 1500 10 15038 1500 10/20 150/75 5/641 1500 15 100 9

Quantitative TestQuantitative Test

The laboratory reports the GM content of the test sample as a percentage value and the calculation basis, e.g. haploid genomes, seeds by number or seeds by mass.

• 22 laboratories submitted quantitative test results


4 5 11 16 17 22 23 27 28 31 32 33 36 37 39 40 44 45 46 54

0.0

1.0

2.0

3.0

4.0

5.0

6.0

GM

O c

onte

nt (%

)

Lab. #

0.2% 2% 4%

PT03: PT03: ResultsResults of of the the quantitative testquantitative testssMean per GMO spiking level and laboratory


0.2% 2% 4%21 0.6424 0.19 2.41 2.6541 0.17 1.90 3.9357 0.32 0.57 2.18

PT03: PT03: ResultsResults of of the the quantitative testquantitative testss


0.0

0.1

0.1

0.2

0.2

0.30.3

0.4

0.4

0.5

0.5

4 5 11 16 17 23 27 28 31 32 33 36 37 40 44 45 46 54

GM

O c

onte

nt (%

)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

4 5 11 16 17 22 23 27 28 31 32 33 36 37 39 40 44 45 46 54

GMO level0.2%

GMO level2%

0

1

2

3

4

5

6

7

8

9

10

4 5 11 16 17 22 23 27 28 31 32 33 36 37 39 40 44 45 46 54

GM

O c

onte

nt (%

)

GMO level4%

ComparisonComparison of of quantitative testquantitative test results results (PT02 and PT03)(PT02 and PT03)


The standard deviation and the variation coefficient are related to the single results and not to the laboratories‘ means.

Test GMO level Mean Standard deviation Variation coeff.

PT03 0.2% 0.19% 0.10% 51%

PT03 2.0% 1.62% 0.92% 57%

PT03 4.0% 3.37% 1.68% 50%

PT02 0.7% 0.73% 0.37% 50%

PT02 1.4% 1.47% 0.91% 62%

PT03: PT03: ClassificationClassification of of the samplesthe samples asas ≤≤ oror > 1% > 1% with the results with the results of of thethe quantitative quantitative testing testing

PT03:

Sample 1 (0.2%): 98.5% classified correctly

Sample 2 (2%):68% classified correctly

Sample 3 (4%):100% classified correctly

PT02:

Sample 1 (0.7%):70% classified correctly

Sample 2 (1.4%):67% classified correctly


<0.2 <0.2 <0.2 0.6 0.7 0.8 1.4 1.8 1.60.09 0.09 0.06 0.7 0.8 0.7 1.8 1.4 1.4<0.1 <0.1 0.1 0.49 0.99 0.87 1.85 1.44 2.180.4 0.1 0.1 0.9 0.7 0.8 1.3 1.5 1.60.15 0.15 0.12 1.03 1.1 1.04 3.18 2.38 2.340.12 0.12 0.12 0.82 0.87 1.43 1.67 1.92 20.05 0.07 0.05 1 0.89 1.04 1.67 1.69 1.890.22 0.2 0.5 1.5 2 0.88 2.9 2.7 2.90.19 0.14 0.19 0.94 1.13 1.06 1.94 3.06 3.54

0.27 <0.1 0.23 0.97 1.47 0.72 3.87 7.88 2.460.16 0.18 0.12 1.65 1.12 1.25 3 2.53 3.110.08 0.08 0.15 1.16 1.33 1.86 3.01 3.18 1.440.14 0.13 0.14 1.57 1.34 1.59 3.27 3.01 3.170.2 0.3 0.1 1.7 1.6 2.3 3.4 3.1 3.60.25 0.18 0.28 2.29 1.83 2.02 4.31 4.54 4.60.27 0.23 0.22 2.13 2.11 2.36 5.07 4.57 4.870.3 0.2 0.2 2.4 2.3 2.7 6.2 5.2 5.10.3 0.3 0.3 2.9 2.9 3.2 5.7 6.5 6.40.18 0.3 0.3 3.91 3.07 2.02 6.22 5.79 5.270.32 0.43 0.17 3.01 4.64 3.97 5.25 6.31 5.25

sample1 (0.2%) sample2 (2%) sample3 (4%)

Summary Summary && ConclusionsConclusions

• There are no serious differences between PT01, PT02 and PT03 regarding the right classification of the samples (qual. Test).

• Despite a variation in reporting false negative and false positive results, the percentage of correctly reported results stays constant.

• Improvement in the sub-sampling test; more similar testing plans and the use of the whole sample for testing.

• Increase in the number of labs which performed the quantitative test. In the quantitative test, the spiking level was underestimated for higher spiking levels. The variation coefficient was similar for all five spiking levels of PT02 and PT03 (50-60%).

• Sample-size, spiking level and events are appropriated for future proficiency tests.


44ndnd Proficiency Proficiency TestTest

• Crop: Soybean

• Small sample number and big sample size (3000 seeds)

• Qualitative, sub-sampling and quantitative Test

• Participation fee for non-ISTA Member

• MTA


Proficiency Test Working Group, GMO TF & Statistics GMO TF

• General concept/aim• Changes according to GM seed testing• Rating system:

qualitative testsquantitative tests

DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on Programme on GM GM Seed TestingSeed Testing


DRAFT:The ISTA Proficiency Test Programme on

GM Seed Testing

• Designing PTs on GM seed testing• Evaluation of Results• Rating laboratories’ performance

! Related to the existing ISTA Proficiency Test Programme! Fits for the purposes of PTs of GM seed testing

DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on Programme on GM GM Seed Testing Seed Testing -- AimsAims


• 2 rounds per year• Programme plan for 1-year period fixed

• Genetic purity testing

• Design varies between the rounds! 10 – 30 samples of the same crop species! At least two GM seeds are introduced in positive seed samples

• Material Transfer Agreement• Lab tests the samples using the methods and procedures

they think are appropriated for the test

DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on GM Programme on GM Seed Testing Seed Testing –– Changes according Changes according to GM to GM seed Testingseed Testing




Total errors for PT3 (12 samples)

Laboratory Number

4 5 7 8 11 12 13 14 15 16 17 19 21 22 23 24 25 26 27 28 29 30 31 32 33 35 36 37 38 39 40 41 44 45 46 47 51 52 54 57

Erro

r fre

quen

cy

0

1

2

3

4


DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on GM Programme on GM Seed Testing Seed Testing -- ExampleExample PT01PT01--PT03PT03

Total errors for PT1, PT2 and PT3 (30 + 10 + 12 = 52 samples)

Laboratory Number for PT1

6 8 11 13 14 18 19 21 23 24 25 27 35 36 40 41 42 43 44 45 46

Erro

r fre

quen

cy

0

2

4

6

8

10

12

14

16

18

20

22

24


3 7 29 35 15 17 32 8 19 47 5 37 25 16 48 34 51 24 27 33 12


44 33 14 45 7 27 31 25 47 12 39 19 17 5 30 35 51 57 13 37 4

f1_total f2_totalf3_total

Is it possible to use the existing system also for GMO PTs?Definitions of the A, B, C and BMP scores


Rating Rating system: qualitative Tests system: qualitative Tests


Score Definition SpecificationA Very good No problem has been detected.

B Good Small problem, but no specific action is suggested to the participant.

C Satisfactory Problems, ISTA indicates there might be actions to consider by the laboratory to explain and correct procedures.

BMP Below minimum performance Poor results, ISTA indicates by a letter that the results were poor and that the laboratory has to try to explain or correct

d

Proposed Rating Decision Rule for Qualitative Testing




Score Decision Rule

A 0 errors

B 1 or 2 errors

C 3 errors

BMP More than 3 errors




Score

Attributed value in a given PT round

Overall score

On a Range of 6 PT rounds for 3 PTs

A 5 A 28-30 14-15B 4 B 21-27 11-13C 3 C 16-20 8-10BMP 0 BMP below 16 <8

Proposed overall rating for all test types based on the in-round rating values Qualitative Testing

DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on Programme on GM GM Seed Testing Seed Testing –– Example Example PT01PT01--PT03PT03


Rating

Score ClassesA B C BMP

Labo

rato

ry fr

eque

ncy

0

5

10

15

20

25

30

35

40

45

PT1 (45 Laboratories)PT2 (50 Laboratories)PT3 (40 Laboratories)PT1, PT2 and PT3 (21 Laboratories)

Rating system: qualitative results

1 lab

Conclusion

• Design and spiking level is practicable as well as the propose rating system

Future

• Finalise the evaluation and rating system for quantitative tests

DraftDraft: ISTA : ISTA Proficiency Proficiency Programme on Programme on GM GM Seed Testing Seed Testing -- AimsAims


Documents

Proficiency Test Working Group, ISTA GMO TF · PT03: Qualitative test A classification (irrespective of the type of method used) of each of the 12 samples as either positive or negative