View
3
Download
0
Category
Preview:
Citation preview
Evaluation Of The Statistics-Based Ames Mutagenicity Model Sarah Nexus And Interpretation Of The Results Obtained
Alex Cayley alex.cayley@lhasalimited.org
Summary
• What is a “statistics-based Ames mutagenicity model” and
why are they important?
• How can we judge how useful these models will be?
(validation statistics, expert interpretation)
• How well does Sarah Nexus (SX) version 1.1 predict and
how can results from the program be interpreted to
increase performance?
• Worked examples of SX version 1.1 predictions
What Are Statistical Models?
• In our sphere no real clear-cut definition but…
• Data for a given endpoint is fed in to the model builder (binary or continuous)
• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values
• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention
• Definition is important for ICH M7 guidance compliance
• May not be as important elsewhere and the distinction is
blurring…
What Are Statistical Models?
• In our sphere no real clear-cut definition but…
• Data for a given endpoint is fed in to the model builder (binary or continuous)
• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values
• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention
• Definition is important for ICH M7 guidance compliance
• May not be as important elsewhere and the distinction is
blurring…
ICH M7 Guidance
What Makes A Good Statistical Model?
7
“right” = Every time Explanation = Full and Reasoned
“right” = Sometimes Explanation = None
Super Expert Scientist
What Makes A Good Statistical Model?
8
“right” = Every time Explanation = Full and Reasoned
“right” = Sometimes Explanation = None
Super Expert Scientist
What Makes A Good Statistical Model?
“right” = Most Times Explanation = Full and Reasoned
“right” = Most Times Explanation = None
“right”= Sometimes Explanation = Some
Expert Scientist
What Makes A Good Statistical Model
QSAR Model
Training Data Validation Expert
Interpretation
1
Pharma A2
Pharma B
3
Pharma C4
Pharma D
Test SetPerformance StatsBA72637275
xx
SEN68386465
xx
SPEC76898085
xx14 Pharma N
Validation of (Q)SAR Models In The Literature
Validation Of Sarah Nexus (v1.1)
Specificity = 69-91% (83% mean) Sensitivity = 38-68% (55% mean) Positive <50
Incorrectly assign 3-4 = ~10%
Balanced Accuracy = Sens + Spec
2 = 62-77% (69% mean)
Prediction Scenarios in Sarah Nexus
Overall Prediction +
Confidence
Hypotheses +
Confidence Positive
Negative
Overruled
Negative Predictions
Negative Predictions
Positive Predictions
Positive Predictions
Confidence Correlation With Predictivity
Equivocal Predictions
Out Of Domain Predictions
An Update
Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM1 879 279 600 77 76 80 74 60 89 188 361 126 46 82 136 22 1212 513 97 416 72 76 64 80 44 90 51 253 65 29 78 84 31 03 4018 576 3442 69 78 56 82 33 92 254 2262 507 198 80 524 272 3134 2862 170 2692 67 83 48 85 17 96 56 1654 283 61 72 621 187 275 4040 725 3315 68 80 50 86 44 89 282 2205 361 278 77 590 343 389
Mean 71 60 81 78
Sarah Nexus V2.0.1
Sarah Nexus V1.2 Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM
1 879 279 600 72 74 68 76 55 85 143 372 115 67 79 136 46 512 513 97 416 74 81 62 86 48 91 44 284 48 27 79 76 34 03 4018 576 3442 67 78 51 82 32 91 231 2261 488 224 80 498 316 1584 2862 170 2692 67 83 48 85 17 96 57 1661 283 61 72 646 192 285 4040 725 3315 62 81 33 90 42 87 186 2444 260 371 81 430 374 243
Mean 68 53 84 78
Conclusions
• Proprietary data sets can give a good indication of the
performance of statistical prediction systems
• Sarah Nexus performs well when tested against a
number of different proprietary validation sets
• Additional information provided for each prediction is also
important in aiding the user to make a final decision
• Negative predictions based purely on negative hypotheses are more reliable
• Positive predictions with a higher confidence are more reliable
Barber et al.; Reg. Tox. Pharm.; 76; 7-20 (2016) http://www.sciencedirect.com/science/article/pii/S0273230015301410
Acknowledgements
• Sandy Weiner
• Joerg Wichard
• Amanda Giddings
• Susanne Glowienke
• Alexis Parenty
• Alessandro Briggo
• Hans-Peter Spirkl
• Alexander Amberg
• Ray Kemper
• Nigel Greene
• Chris Barber
• Thierry Hanser
• Alex Harding
• Crina Heghes
• Jonathan Vessey
• Stephane Werner
Questions?
Work in progress disclaimer
25
This document is intended to outline our general product direction and is for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon. The development, release, and timing of any features or functionality described for Lhasa Limited’s products remains at the sole discretion of Lhasa Limited.
Recommended