DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The Challenges of...
Preview:
Citation preview
- Slide 1
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com The Challenges of Validating R
Managing R in a Commercial Environment
- Slide 2
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What does validation mean Why
validate R The challenges of validating R Validation tools ValidR
Questions Agenda
- Slide 3
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What does validation mean?
Establishing documented evidence which provides a high degree of
assurance that a specific process will consistently produce a
product, meeting its predetermined specifications and quality
attributes U.S. Food and Drug Administration (2013). 21 CFR Part
11: Electronic Records, Electronic Signatures,
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1
- Slide 4
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Why validate R? No guarantee
that packages meet specifications Must comply with industry
regulations to be used in regulated environments
- Slide 5
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Why validate R? R Foundation
response to CFR Part 11: R: Regulatory Compliance and Validation
Issues A Guidance Document for the Use of R in Regulated Clinical
Trial Environments 1 Document relates to only base R and
recommended packages R CMD CHECK provides no guarantee that an R
add-on package meets its specifications Authors do not have to
write tests 1. http://www.r-project.org/doc/R-FDA.pdf
- Slide 6
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com ## WARNING: ## This code is a
complete hack, ## may or may not work, etc.. ## Use your own risk.
You have been warned. The recommended R package: codetools, version
0.2-8. Why validate R?
- Slide 7
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What is validation? Establishing
documented evidence which provides a high degree of assurance that
a specific process will consistently produce a product, meeting its
predetermined specifications and quality attributes U.S. Food and
Drug Administration (2013). 21 CFR Part 11: Electronic Records,
Electronic Signatures,
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1
- Slide 8
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com The challenges of validating R
Define Requirements Build Software Test Software Against
Requirements Create Documentation Typical Software Development
Process Investigation Remediation Certification
- Slide 9
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Challenge 1 : Defining
requirements Packages dont typically come with a list of
requirements How do we determine the intended use? Package
descriptions and help files Package vignettes Previous experience
of using the packages
- Slide 10
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Challenge 2 : Understanding a
package Software is already built Source code is available How do
we gain an understanding of the package structure? The functionMap
package
- Slide 11
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com The functionMap package
Describes the relationship of functions within an R package Parse R
code Create dynamic graphics of function relationships Create a
network object to show relationships Creates an interactive graphic
See which functions a given function calls Determine functions that
are called within each function
- Slide 12
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com
- Slide 13
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com
- Slide 14
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Challenge 3 : Testing the
requirements Package authors dont have to write tests How do we
test the requirements? Writing specific unit tests for requirements
Understand the level of testing with testCoverage
- Slide 15
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com The testCoverage package
Determines which code in a package is covered by unit tests Reads
package code and adds tracepoints Runs package unit tests and marks
points as hit Creates a full report of the code hit
- Slide 16
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com
- Slide 17
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Challenge 4 : Validation
documentation Creating consistent documentation for each R package
Automatic report template generation knitr Audit trail
- Slide 18
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Challenge 5 : Deployment
Installation Qualification (IQ) Operation Qualification (OQ)
Performance Qualification (PQ)
- Slide 19
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com IQ - Is everything installed as
expected? Perform an audit of the installed files OQ - Does
everything operate as expected? Run the tests used in validation
locally PQ - Does everything perform as expected? Measure the
performance of the installed software VALIDATION Qualification
process
- Slide 20
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What is ValidR? R (currently
3.0.2) Core and Recommended Packages 60+ additional packages
Validation Packages Validation Documentation Windows Installer
Linux RPMs
- Slide 21
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What are the benefits? Industry
Gives analysts the tools they want use Provides assurance to the IT
department Reduces risk and cost to business managers Accelerated
analytical cycle R community Bug fixes patches Increased user
base
- Slide 22
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com The ValidR roadmap September
2014 EARL Conference Release testCoverage and functionMap on CRAN
Easter 2015 Release of validated R 3.1.x Summer 2015 Release of
validated additional packages
- Slide 23
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com What does validation mean
Assurance, evidence Why validate R Provide assurance, evidence The
challenges of validating R Validation tools ValidR Methodology,
software, documentation Benefits Summary
- Slide 24
- DRAFT Richard Chandler-Mant R Consultant
rchandler-mant@mango-solutions.com Questions?