Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
1
IVT EU CSV 2010
Automate Your CSV ProcessAutomate Your CSV ProcessTemplates, Tools & TricksTemplates, Tools & TricksTemplates, Tools & TricksTemplates, Tools & Tricks
Tyson M. Mew, PresidentOfni Systems Inc.
About the Speaker
Ty MewTy Mew is President of Ofni Systems, Inc., a is President of Ofni Systems, Inc., a software provider and Part 11 consulting firm Hesoftware provider and Part 11 consulting firm Hesoftware provider and Part 11 consulting firm. He software provider and Part 11 consulting firm. He has a strong background in software development has a strong background in software development and computer system validation, and has spoken at and computer system validation, and has spoken at many conferences and workshops on Part 11. Ty many conferences and workshops on Part 11. Ty regularly conducts training sessions for regularly conducts training sessions for organizations to educate employees about the organizations to educate employees about the specific requirements of the rule and how tospecific requirements of the rule and how to
October 1, 2010 2
specific requirements of the rule and how to specific requirements of the rule and how to incorporate it into daily practices. He is an active incorporate it into daily practices. He is an active member of ISPE, PDA, SQA and the Triangle member of ISPE, PDA, SQA and the Triangle PEERS group.PEERS group.
2
Getting Started
Why CSV is well-suited to iautomation
Formalize your strategyPreparing templates for documents and certain types of systemsyp yDetermine and execute your testing strategy
3
Quick Poll
Are you primarily validating:• Commercial Off-The Shelf (COTS) without access
to code (black box testing)• Open systems with access to system code (white
box testing)Do you execute protocols electronically?y p y
Automating Computer System Validation
Software is Well-Situated to Automation
Use automated software tools to • Analyze Software• Detect Objects (Screens, Reports, Code)• Identify Missing Objects• Create test scripts
4
Formalizing Validation Strategy
Create a General Strategy for All Projects:• Identify Validation Tactics/Testing Approaches for
Common Elements– Data Inputs– Formulas, Calculations– Reports, Charts
fi i f ifi f• Define Strategies for Specific Software TypesValidation Strategy Documented in Validation Master Plan
Formalizing Validation Strategy
Unifies Validation ApproachEnsures Internal Agreement with Validation StrategyImproves Training of Validation Professionals Facilitates Review/Approval ProcessProvides chance to implement best practices for lean project management
5
Validation TemplatesTemplates Improve document consistency and
efficiency of document creationefficiency of document creationTemplates Include:• SOP identified Sections (Objectives, Scope, etc.)• Instructions for completing specific sections• Template text with variable names• Accepted Formatting
Hint: Fewer templates are better than more templates. Use variables to create constant text.
Automated Document Generation
Extension of TemplatesIncludes specific verbiage with variables.
Variables are replaced with specific data.Requirement lists automatically inserted into appropriate sections of validation documentsThi h k ll ith i ilThis approach works well with similar
programs with similar requirements (Groups of spreadsheets, etc.)
6
Example Variable NamesSystem NameProgram Version NumberProgram Version NumberFile Name, Data File Name, Security File Name, External Code Library NameFile LocationCompany Name, Company AddressOwning Department System OwnerOwning Department, System OwnerDocument Names, ID#s, Abbreviations, TitlesProtocol Names, Abbreviation
Determining Testing Strategy
D t i h t i t d thDetermine what you are going to do, then execute your planValidation Professionals test according to planQuality review/approve documents based on existing standards
7
Automate the Generation of CSV Documents
Breaking down systems into discrete lid t bl bj tvalidatable objects
Collecting testable requirements Integrating Risk Assessments into the processGenerating Design Specifications directly from
dsource codeAutomatic generation, tracking and updating of the Requirements Traceability Matrix
Breaking Down Systems intoDiscrete Validatable Objects
fi i i f hDefine Distinct Parts of the Program•Forms, Reports•Worksheets, Charts•Workflows
Automatically Create List of Validatable ObjectsNote: We use software to analyze programs, record logical order for forms and reports. Software also identifies missing objects
8
Collecting RequirementsDefine requirements during project walk-through
Additional requirements can be gathered from key end users, developers, system owners.
Note: GAMP 5 specifically allows combining/separating of specification documents to facilitate validationfacilitate validation.
Hint: Use software which allows you to record requirements as the validatable objects are recorded
Integrated Risk AssessmentValidation Resources are limited; a Risk-Based Validation approach is necessary to focus limitedValidation approach is necessary to focus limited resourcesRisk Assessment is performed at the requirement level• Functional Specification• Design Specification
A k D l /K S t U /S t OAsk Developers/Key System Users/System Owners where system risk is. They know and you usually get a better ROI than trying to analyze every requirement.
9
Define Testing for Each Object
Forms: Input, Processing, Output, SecurityReports: Data, Calculations, FormattingSpreadsheets: Data Entry, Calculations, Data Outputs, LabelsCharts: Data Ranges, Calculations, L b l /F ttiLabels/FormattingWorkflows: Process Flow
Generating Design Specifications
Once requirements are defined, use software t l t l bj t d d it bltools to analyze objects and produce suitable outputOnly applicable to open source code.
Hi t U ft i t t lHint: Use software scripts to analyze common objects and produce suitable content for the Design Specification.
10
Example of Generating Design Specifications: Data Input Form
Examples of Generating Design Specifications: Spreadsheet Calculations
11
Automatic Requirement Traceability
Several Programs exist which will t ti ll t th R i tautomatically generate the Requirement
Traceability MatrixUpdates if Requirements are Moved/EditedDisplays if Functional Requirements exists without corresponding Design specificationwithout corresponding Design specificationDisplays if Requirements exist which do not have a defined Test Case
12
Test Cases and Test Plans
Testing based on Risk AssessmentsInput Testing vs. Challenge Testing, Workflow TestingAutomating with micro-test casesAuto-creation of test scripts from your requirementsrequirementsCreate specific test cases for use with change control
How Much Is Enough?Automating Testing allows more time for Testing, Less Time spent on DocumentationLess Time spent on Documentation.Defining specific requirements based on Risk allows for use of different templates based on specific functional needs.Automated creation of Test Cases/Protocols allows validation resources to focus on high risk sections of
f i liprogram functionalityRequired business functionality should receive special attention.
13
Input vs. Challenge TestingHigh Risk Fields Should have Challenge Tests
Status Fields • Do not accept <NULL>• If Drop-Down List Used, Status not accept other values• Default Value should not be Pass, Success, etc.
Dates• Do not accept clearly incorrect dates• Do not accept unordered dates (Date of Test before Date
Received, etc.)
Workflow Testing
Extremely useful technique for systems with a d fi d kfl (S ldefined process or workflow (Sample Management, Manufacturing Process, etc.)• Verify that entered data can move through entire
workflow• Verify that fail-points prevent data from moving
forward• Verify that data cannot be accessed from
inappropriate parts of the workflow
14
Using Micro-Test CasesMicro-Test Cases are test cases for common requirements have pre written verbiage (withrequirements have pre-written verbiage (with variables) which is inserted into appropriate sections of the document.Parallel sections inserted into Requirement Documents, Design Documents and Testing DocumentsDocumentsSmaller test cases combined into larger test cases
Using Micro-Test CasesMicro-Test Cases most useful for common, repeated test stepstest steps.• Adding/Closing Forms or Reports• Data Entry into Fields• Screen Navigation• Printing, Generating Charts, Etc.• Confirmation of Formulas/CalculationsConfirmation of Formulas/Calculations• Any place in Test Protocol where similar operations are
repeated with similar resultsContent of Micro-Test case modified with variables
15
Enter Required Variables
Lines of Test Protocol Generated
16
Automatic Generation of Test Protocols
In certain systems, software can be used to automate the generation of entire test protocols based on defined inputsData Entry ScreensIndividual Spreadsheet WorksheetspSecurity Testing
Automatic Generation of Test Protocols
17
Test Cases for Change Control
Certain Test Cases Are Particularly Useful h if i f ti lit ft Chwhen verifying functionality after Change
Control• Regression Testing• Functionality Identified as High RiskWherever possible Test Cases should beWherever possible, Test Cases should be written mindful of reuse during Change Control
Protocol Execution, Documentation and Managing Deviations
Paper vs. Electronic executionCapturing actual results and screen shots for maximum effectivenessGenerating, processing and closing test and script deviations
18
Paper vs. Electronic Execution
Electronic Protocol ExecutionR l t diti l d t h i• Replaces traditional pen and paper techniques
• Just as word processors improve writing, electronic protocol execution improves paper executions
Documents can be exported and reviewed by traditional means (“Type-writer Rule”)
N El i P l S f bNote: Electronic Protocol Software must be compliant with 21 CFR 11, with audit trails, security, electronic signatures, etc.
Example of Electronic Protocol Execution
19
Documenting Protocol Execution with Screen Shots
As each step is completed, collect screen shotInsert Screen shots automatically into finished protocoly pConfigure Screen shots to include user name, date/time, etc. which further document testing
Hint: We use screen shot software, integrated with Electronic Protocol Execution software. This significantly reduces time organizing and numbering screen shots.
Warning!Screen shots do not “prove” a test case was successful, only
provides evidence that a test was performed.
Documenting Protocol Execution with Screen Shots
20
Automating Deviation Reporting
When a test step fails, deviation is t ti ll t dautomatically created
User identifies deviation type, based on this determination, some fields are automatically filled.Deviations can be viewed in Real-TimeDeviations can be viewed in Real-TimeDeviations can be resolved according to each companies procedures
21
Validation Case Study ReviewsEnsure compliance with 21 CFR 11• Technological• Procedural
Risk AssessmentGather Requirements• Screen Specific• Workflow Specific
October 1, 2010 41
• SecurityWrite Requirement DocumentsTesting Documents
Case StudiesSpreadsheets for CRO
A li d t l t d h t t id• Applied tools to spreadsheets to provide technological controls, including audit trails, user level security and electronic signatures
• Spreadsheet requirements generated from micro templates.
• Requirements verified through protocol executionq g pResult: Entire spreadsheet library quickly and
efficiently validated
22
Case StudiesEngineering system for Pharmaceutical Manufacture
R i t th d f k d d l• Requirements gathered from key users and developers• Software emphasized moving through exact workflows, so
testing plan modified to focus on workflows over data entry screens with associated reports
• Requirements verified through protocol execution
Result: Emphasized workflows over input/output model when appropriate, which improved testing and overall validation effort.
Case StudiesElectronic Document Capture System• Worked with developers to demonstrate compliance with 21 W p p
CFR 11• Program used Client/Server model, with digital pen inputs• Extensive use of virtual machines to model each section of
program – different OS, different sections• Tested specific form events• Requirements verified through protocol execution
R lt V lid ti ff t ll d i k d ffi i tResults: Validation effort allowed quick and efficient compliance with 21 CFR Part 11 regulations. After the validation process was complete, the validation results were reviewed and approved by a third party auditor
23
Case Studies
Purchasing System for Pharmaceutical M f tManufacturer• Requirements gathered from company SOP and
instructions• Worked with 3rd party vendor to add security• Met with both End users and developersp• Requirements verified through protocol execution
Result: Third party software made compliant with 21 CFR 11, with implemented Security requirements
Case Studies
Clinical Research Database from CRO• Clear requirements, no requirement drift• Used existing templates, validation document
generation and electronic protocol execution• Requirements verified through protocol execution
Result: Validation completed in 12 business days.
24
Live Examples
Demonstration of the concepts described in this presentation
Thank You!
Questions? Tyson M. Mew(919) 844-2494Ofni Systems [email protected]
October 1, 2010 48
www.OfniSystems.comReferences and additional materials will be available at
www.OfniSystems.com at the conclusion of the conference.
25
AdditionalAdditional Reference M t i lMaterial
Solutions for Common Software
Web Based Applications Custom Databases (MS Access, FoxPro, SQL Server, Oracle, etc.)Spreadsheets (MS Excel, Quattro, Lotus, etc.)Connections to LIMS
In addition to system specific issues, all systems must demonstrate control of data, and meet any applicable business requirements.
26
Web Based Applications
Define System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/ComputationsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security RequirementsMake sure all functions are testedMake sure all functions are tested
Custom DatabasesAccess, FoxPro, SQL Server, Oracle, etc.Access, FoxPro, SQL Server, Oracle, etc.
Define System InterfacesDefine System InterfacesDefine System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/Computations
October 1, 2010 52
Define Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security Requirements
27
Spreadsheets
Excel, Quattro, Lotus, etc.Define Data Input Cells/RangesDefine Spreadsheet Processing• Calculation Cells• Internal FunctionsDefine System Outputs/ChartsDefine Security Requirements
Connections to LIMS
Define Data Type and RequirementsDefine Data Workflow• Define Data Input to LIMS • Define Output from LIMS• Calculations are performed inside p
LIMS and exported• Calculations are performed outside
LIMS and re-imported
28
Gathering Requirements
Interviews with:• Key System End-Users• System Owner• Developer
Good communication between the validation teamGood communication between the validation team and all end users improves all aspects of the project and adds more value to the project.
Gathering RequirementsScreen Specific
I t t t• Input tests– Make sure that all fields accept appropriate
data entry– Use challenge tests where appropriate
• Processing– Make sure all functions (buttons, etc.) operate as
expected– Make sure all calculations produce correct results
29
Gathering RequirementsWorkflow Specific
F ll ifi d t t th h t th kfl• Follow specific data entry throughout the workflow.• Follow data transitions in workflows
Security• Define specific groups
fi ifi kfl il bl• Define specific workflows available• Define specific functionality• Test what users can and cannot do
Risk Assessment
Determine what parts of the software create the t i k d f t ti thmost risk and focus testing there.
Risk Assessment should be multi-disciplinary, and should include validation professionals, key end users, quality review, developers and system owner.system owner.Focus on known areas first