Evaluation Of The Statistics-Based Ames Mutagenicity Model Sarah Nexus And Interpretation Of The Results Obtained

Alex Cayley alex.cayley@lhasalimited.org

Summary

• What is a “statistics-based Ames mutagenicity model” and

why are they important?

• How can we judge how useful these models will be?

(validation statistics, expert interpretation)

• How well does Sarah Nexus (SX) version 1.1 predict and

how can results from the program be interpreted to

increase performance?

• Worked examples of SX version 1.1 predictions

What Are Statistical Models?

• In our sphere no real clear-cut definition but…

• Data for a given endpoint is fed in to the model builder (binary or continuous)

• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values

• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention

• Definition is important for ICH M7 guidance compliance

• May not be as important elsewhere and the distinction is

blurring…

What Are Statistical Models?

• In our sphere no real clear-cut definition but…

• Data for a given endpoint is fed in to the model builder (binary or continuous)

• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values

• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention

• Definition is important for ICH M7 guidance compliance

• May not be as important elsewhere and the distinction is

blurring…

ICH M7 Guidance

What Makes A Good Statistical Model?

7

“right” = Every time Explanation = Full and Reasoned

“right” = Sometimes Explanation = None

Super Expert Scientist

What Makes A Good Statistical Model?

8

“right” = Every time Explanation = Full and Reasoned

“right” = Sometimes Explanation = None

Super Expert Scientist

What Makes A Good Statistical Model?

“right” = Most Times Explanation = Full and Reasoned

“right” = Most Times Explanation = None

“right”= Sometimes Explanation = Some

Expert Scientist

What Makes A Good Statistical Model

QSAR Model

Training Data Validation Expert

Interpretation

1

Pharma A2

Pharma B

3

Pharma C4

Pharma D

Test SetPerformance StatsBA72637275

xx

SEN68386465

xx

SPEC76898085

xx14 Pharma N

Validation of (Q)SAR Models In The Literature

Validation Of Sarah Nexus (v1.1)

Specificity = 69-91% (83% mean) Sensitivity = 38-68% (55% mean) Positive <50

Incorrectly assign 3-4 = ~10%

Balanced Accuracy = Sens + Spec

2 = 62-77% (69% mean)

Prediction Scenarios in Sarah Nexus

Overall Prediction +

Confidence

Hypotheses +

Confidence Positive

Negative

Overruled

Negative Predictions

Negative Predictions

Positive Predictions

Positive Predictions

Confidence Correlation With Predictivity

Equivocal Predictions

Out Of Domain Predictions

An Update

Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM1 879 279 600 77 76 80 74 60 89 188 361 126 46 82 136 22 1212 513 97 416 72 76 64 80 44 90 51 253 65 29 78 84 31 03 4018 576 3442 69 78 56 82 33 92 254 2262 507 198 80 524 272 3134 2862 170 2692 67 83 48 85 17 96 56 1654 283 61 72 621 187 275 4040 725 3315 68 80 50 86 44 89 282 2205 361 278 77 590 343 389

Mean 71 60 81 78

Sarah Nexus V2.0.1

Sarah Nexus V1.2 Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM

1 879 279 600 72 74 68 76 55 85 143 372 115 67 79 136 46 512 513 97 416 74 81 62 86 48 91 44 284 48 27 79 76 34 03 4018 576 3442 67 78 51 82 32 91 231 2261 488 224 80 498 316 1584 2862 170 2692 67 83 48 85 17 96 57 1661 283 61 72 646 192 285 4040 725 3315 62 81 33 90 42 87 186 2444 260 371 81 430 374 243

Mean 68 53 84 78

Conclusions

• Proprietary data sets can give a good indication of the

performance of statistical prediction systems

• Sarah Nexus performs well when tested against a

number of different proprietary validation sets

• Additional information provided for each prediction is also

important in aiding the user to make a final decision

• Negative predictions based purely on negative hypotheses are more reliable

• Positive predictions with a higher confidence are more reliable

Barber et al.; Reg. Tox. Pharm.; 76; 7-20 (2016) http://www.sciencedirect.com/science/article/pii/S0273230015301410

Acknowledgements

• Sandy Weiner

• Joerg Wichard

• Amanda Giddings

• Susanne Glowienke

• Alexis Parenty

• Alessandro Briggo

• Hans-Peter Spirkl

• Alexander Amberg

• Ray Kemper

• Nigel Greene

• Chris Barber

• Thierry Hanser

• Alex Harding

• Crina Heghes

• Jonathan Vessey

• Stephane Werner

Questions?

Work in progress disclaimer

25

This document is intended to outline our general product direction and is for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon. The development, release, and timing of any features or functionality described for Lhasa Limited’s products remains at the sole discretion of Lhasa Limited.