Embed Size (px)
Citation preview
Implementing the New Part 11 Guidance 1
Infrastructure Qualification
Jerry AndersonIsis Pharmaceuticals
AGENDA
The Case For Qualifying Your Infrastructure
Qualification Foundations
Risk Management
Qualification Methodology
Background
1990’s: Increased regulatory scrutiny on all things computer-related
1997: Part 11
1999: GAMP article “Compliance for Corporate Information Technology,” Pharmaceutical Engineering
2001: Pharmacia warning letter
2002: IVT Infrastructure Qualification Conference
2004: Article “IVT Network Infrastructure Qualification Proposed Standard,” Journal of Validation Technology
2005: “GAMP Good Practice Guide: IT Infrastructure Control and Compliance”
Pharmacia Warning Letter
A custom networked application and an off-the-shelf networked application
– No revision control system
– Failure to update and maintain structural and functional diagrams and design descriptions
– Failure to update and maintain diagrams with text descriptions identifying interfaces to other network programs
– Inadequate standard operation procedures to ensure that records are included with validation documentation, maintained and updated when changes are made
– The network was not included in the validation efforts and therefore lacked adequate documentation controls
Translation
“Okay, your distributed application worked during the time you ran your application testing. What assurance do you have that it would
have worked an hour later? A day later? A year later?”
“Do you have any controls over your network and its services?”
“No?”
“You fail.”
The life sciences world went nutzo
Consultants, however…
What Is Infrastructure?
Network hardware, appliances, software, and services
“Applications”: Active Directory, enterprise backup, network/application monitoring…
Data center facilities
Servers, operating systems, data management software
Clients (PCs, tablets) and their software
What Is Infrastructure: Another Perspective
An application is software doing useful work
– A GxP application is software that automates a regulated process, and/or creates and manipulates regulated data
Infrastructure is the hardware, software, communications, and facilities that provide the operating environment for applications
– GxP infrastructure supports applications and ensures a high degree of integrity for the supported data
Why Qualify Infrastructure?
To generate documentation proving it is well-built and in a state of control
To provide a qualified home for validated applications
To satisfy inspectors
To prevent regulatory action
These are different ways of saying
“Because we’re forced to!”
Regulatory Risk
Fairly low
– No requirement in the law
– cGxP expectation based on regulatory action, FDA statements regarding distributed applications on a network, implications in Annex 11
– We’ve yet to see much about companies being cited for not having qualified their infrastructure
– It’s never come up in my inspections
Inspectors open the first door (validated applications)
If they like what they see, they tend to find another first door
If they don’t like what they see, they walk farther in and open the next door
Why Qualify Infrastructure?
Better reasons:
– Provide better documentation of functionality
– Make IT knowledge transfer easier
– Lower the average downtime of applications
– Reduce the number of network outages
– Reduce the number of trouble/incident tickets
– Reduce mean time to troubleshoot/repair
– Increase the success rate of planned changes
– Lower the overall cost of providing IT services
Is Your Infrastructure In A State Of Control?
If it’s not, maybe it should be:
“Problems arising from system misconfigurations
are the largest source of network failures, system
downtime, help desk calls and security breaches.”
- Network World
Bottom-Line
There’s more business risk than compliance risk in your IT infrastructure
– The compliance risk of most infrastructure is lower than your lowest-risk GxP application
Therefore, the need to get infrastructure into a state of control is driven more by business need than compliance need
– GxP and Sarbanes-Oxley regulations are driving companies to put IT controls in place, but all companies would benefit from better IT infrastructure documentation and management
Qualifying your infrastructure reduces business and compliance risks at the same time
AGENDA
The Case For Qualifying Your Infrastructure
Qualification Foundations
Risk Management
Qualification Methodology
Validation vs. Qualification
Application Validation
– Ensures that an application was specified, designed, built, and deployed following good engineering and quality practices; that it meets the specified user requirements; and that it appropriately automates the GxP process / GxP data manipulation
Infrastructure Qualification
– Ensures that infrastructure meets specified design and configuration requirements (if any) and is maintained in a state of control
Infrastructure qualification is all about good engineering and quality assurance
Good Engineering & Quality Assurance
The GAMP infrastructure good practice guide stresses these
IT infrastructure (like anything else) should be designed, deployed, and maintained using good engineering practices (GEP)
The independent quality unit uses quality assurance practices to:
– Verify GEP and other compliance requirements during qualification activities
– Verify ongoing GEP, compliance, and control
Hardware and Software Standards
IVT:
– “Standards form the basis for controlled, uniformly applied technical solutions… This, in turn, is the first step in providing a qualified network infrastructure.”
– “Standards should be available for servers, operating systems, database management software, desktop workstations, base desktop utilities, network hardware, network operating systems, and other key infrastructure components.”
GAMP:
– “If standard platform components, such as standard server and client configurations, are adequately managed, the initial qualification of the platform component becomes a standard qualification package which permits efficient cost-effective duplication…”
Staff Qualifications
IVT and GAMP both stress the following:
– Documented roles and job requirements
– Documented experience and qualifications
– Documented training on relevant procedures
GAMP goes on to state that a formal quality management system which enforces these requirements can make the infrastructure qualification job easier
Supplier Management
Supplier management is required throughout the lifecycle of systems and infrastructure
– Before purchase: vendor assessment
– Management of consulting services during development and deployment
– Management of service contracts, bug fixes, and version upgrades
– Controls on outsourcing of applications, services, infrastructure management
– Management of infrastructure-specific services (e.g. off-site data media storage, disaster recovery services)
Configuration Management
A key (maybe THE key) to successful infrastructure management
– Configuration identification, control, status accounting, auditing
Proper configuration management during development is essential for knowing what you’re qualifying
– Done right, it forms a big part of your documentation effort
After qualification is completed, 50%+ of the work required to maintain the qualified state is configuration management work
AGENDA
The Case For Qualifying Your Infrastructure
Qualification Foundations
Risk Management
Qualification Methodology
Infrastructure Risk
Key concept: IT infrastructure has low residual risk compared to validated applications
– Infrastructure doesn’t automate a regulated process; instead, it helps keeps applications running and talking to each other, and helps protect data availability, integrity, and confidentiality
– That’s why we’re talking qualification, not validation
– In fact, what we’re REALLY talking about is good engineering and quality assurance practices
Still, the risk management process can be useful in determining what and what not to qualify, and how much to do
Risk Management
The ISO risk management process:
1. Risk assessment
a. Risk analysis: identifies types and levels of risk
Impact * probability of occurrence * probability of non-detection
b. Risk evaluation: categorizes identified risks as acceptable or not
2. Risk control: implements controls to reduce risks to acceptable levels
3. Periodic review and evaluation: captures missed or changed risks
1. Risk Assessment
Risk assessment should be used at the beginning of the qualification effort to:
– Identify what elements are in scope, based on their risk
– Identify the extent of each element’s qualification
Let’s look at one risk on an example infrastructure element: a virtual private network (VPN) appliance
1a. Risk Analysis
Example element: VPN appliance
– Hazard: allows unauthenticated user onto network
– Impact: *HIGH*
– Probability of occurrence: *LOW*
– Probability of non-detection: *MEDIUM*
1a. Risk Analysis
*Low* Med High
*High* *Med* High High
Med Low Med High
Low Low 3 Med
Probability of
Occurrence
Imp
act
Risk Classification
Determine Risk Classification:
1. Determine the probability of the bad thing occurring
2. Determine the impact if the bad thing occurs
3. Plot the risk exposure:
– The Impact if it occurs is *HIGH*
– The Probability of occurrence is *LOW*
– Our Risk Classification is *MEDIUM*
1a. Risk Analysis
Low *Med* High
High Med High High
*Med* Low *Med* High
Low Low Low MedRis
k C
lassif
icati
on
Risk Priority
Probability of
Non-Detection
Determine Risk Priority:
1. Copy the risk classification from the previous step
2. Determine the probability of not detecting an occurrence
3. Plot the risk priority:
– Our Risk Classification was *MEDIUM*
– The Probability of Non-Detection is *MEDIUM*
– Our overall Risk Priority for this risk is *MEDIUM*
1b. Risk Evaluation
Example element: VPN appliance
– Is a medium risk of an unauthorized individual being allowed onto your network acceptable to you?
– If yes: you’re done managing this risk
– If no: you’ll need to implement controls to mitigate this risk
Our answer: Unacceptable Risk
2. Risk Control
Example element: VPN appliance
Unacceptable risk: lets unauthenticated user onto network
Possible controls
– General: put VPN appliance “in scope” for qualification
– Specific:
Integration of VPN appliance and authentication server
Testing
Configuration management/change control
SOPs, work instructions, training
Supplier assessments
Note
This example was very specific
A best practice is to automatically put any element that protects your network “in scope” for qualification
Examples:
– Authentication services
– VPN services
– Firewalls
– Virus protection
– Intrusion detection
3. Periodic Review
On a periodic basis, reexamine risks and controls
– Were all identified risks estimated properly?
– Are controls still in place and sufficient?
– Are any new risks present?
AGENDA
The Case For Qualifying Your Infrastructure
Qualification Foundations
Risk Management
Qualification Methodology
Overview
The minimum
– Partnership with IT management
– SOPs
– A quality management system within IT
– GxP training for IT
– Periodic audits by the quality unit
A step up
– Formal qualification documentation for key infrastructure elements
Partnership With IT Management
IT is driven to provide and change services quickly
Quality and validation groups tend to slow things down and create more work
These two things often don’t mix well
One way to deal with this is for the quality group to be tyrannical
A better approach is to build partnership and trust
– When IT understands the risks, and Quality bends (but not breaks) to support the business needs, that’s partnership
Only IT management can make IT compliance happen
General Quality SOPs
Training
Supplier management
Document and record management
Validation / qualification
Change control
Deviations / CAPA
IT SOPs
Security
Backup / restore / archive
Incident / problem management
Database administration
Business continuity / disaster recovery
System administration
Internal IT Quality System (work instructions)
Platform management (general)
– New hardware/software
– Configuration and change management
– Preventive maintenance and problem resolution
– Service startup, shutdown, restrictions
– System monitoring, event/problem logging, problem tracking
– System retirement, data archival/retrieval
Internal IT Quality System (work instructions)
Servers and mainframes: job scheduling
Network management
Client management
– Creation and management of standard image
– Software distribution/upgrade
– Virus protection
Security
– Physical security/access
– Account management, password security, access rights
– Administrator accounts, intrusions, vulnerabilities
Internal IT Quality System (work instructions)
Data management
– Backups, restores, media management
Quality management
– Service level agreements
– Risk management
Continuity management
– Disaster recovery
– Contingency planning
GxP Training For IT
People tend to be more willing to comply if they understand the risks
“Read and understand” training on SOPs is good
General GxP training on regulatory expectations and the consequences of non-compliance is better
Specific training on regulatory expectations regarding validated applications and IT infrastructure is outstanding
Periodic Audits By Quality
IT should be a regular component of the internal audit program
The first audit could be considered a “qualification audit”
– Are we good to go? Is IT self-managing for compliance?
Future audits verify continued compliance, perhaps to changing regulatory expectations, and verify improvement / corrections
Now The Minimum Requirements Are In Place
Quality is partnered with IT management
High level SOPs are in place
Lower level quality management system within IT is in place
IT is GxP-trained
Quality is auditing IT (maybe annually)
Is that enough?
Probably.
The Typical Inspector
He opens the door, probably on the inventory of GxP equipment and applications
Next door: a validated distributed software application
If he likes what he sees, he’ll probably move on to the labs, or batch records, or lunch
But he may open the IT infrastructure door
Your Answer
“Here are the SOPs in place…”
“Here are the work instructions governing IT’s day-to-day operations…”
“Here are the GxP training records for IT personnel…”
“Here is the 2015 internal audit schedule showing an audit of the IT quality management system…”
Nothing to see here… Move along…
Don’t Bite Off More Than You Can Chew
Many companies don’t go even this far
Others do, but don’t have the resources to go further
If you can go further, and want to… “Qualification Plus”
Remember: Qualification, Not Validation
There’s no regulated process being automated, so you typically don’t need a user requirements specification
– “The router must route? The switch must switch?”
No URS means limited functional requirements
– Resist the urge to quote IETF RFCs...
This means we can skip right to design specifications
Then execute IOQs to verify minimums and record installations
No need to PQ devices/software when there’s no user process
– But you might want to PQ the entire network
General Qualification Approach
Qualification plan
Design specs (with any functional requirements and diagrams)
Installation/Operation Qualifications (IOQs - one per infrastructure element or per logical grouping)
Bonus: a network Performance Qualification (PQ)
Qualification report
SOPs and work instructions
Qualification Plan
Describes high-level approach
Says what’s in scope
– E.g. parts of Active Directory are “in” but Exchange and SMTP are “out”
Required activities, documents, other deliverables
Roles and responsibilities
Design Specification
Any functional requirements can be captured here
Document what your “minimums” and configurations are
You might have just one design spec covering your entire network (ISO layers 1-3)
Possibly other design specs covering specific GxP services:
– AD authentication and authorization (group policies)
– NTP
– VPN / IPSEC / TLS
– Directories other than AD
Critical non-GxP services? (e.g. DHCP)
Design Specification
A design spec can apply to a single device, a family of devices, or an entire network
Recommended approach:
– A design spec for the network
– A design spec for GxP data center(s)
– A design spec for GxP server farm, SAN, etc. (combined as appropriate)
– A design spec for each major GxP network service (e.g. Active Directory)
– A design spec for desktop clients
Design Specification
Each design specification describes:
– An element of your infrastructure (e.g. routers)
– How that element is used/what it must do
– Any minimum functional or configuration requirements
It’s OK to base your design on what you already have
– List your existing minimums as your requirements (e.g. minimum RAM, minimum IOS version, minimum circuit bandwidth)
Include vendor recommendations for installation and configuration as appropriate
IOQ
Challenges/records installation and configuration data
Recommended approach:
– One IOQ for the network
If you decide to do an IOQ per equipment type (router, switch, PC):
– Execute the full operational challenge for one family member
– Record the installation for every member
– Refer to IVT’s ‘cookie cutter’ approach for PCs, and GAMP’s ‘horizontal’ approach to equipment types
– One IOQ per GxP network service
– One IOQ per GxP data center
IOQ
If you’re doing IOQs for network equipment, focus on the “I”
– For device families: do functional challenges once (i.e. OQ just one router); do installation documentation for all instances of the model/family
– Also: think about making your functional challenges “bench qualifications.” Examples:
New Cisco router: you’re pretty confident that it routes datagrams, so just document that what you buy meets your specifications for the role that the router will play on your network
Existing router: you already know it’s working, so just document its installation and configuration information
This means that you might be able to IOQ most or all of your network equipment from a central location (e.g. using CiscoWorks, OpenView)
IOQ
Focus more on the “O” for network services
– Low-level services (e.g. DNS, DHCP) = small “o”
– Higher-level, record-critical, or security-critical services (e.g. Active Directory, NIS+, LDAP, VPN, firewall, data backup) = BIG “O”
Infrastructure Elements to Challenge With IOQs
Network devices
Network services
Data centers
Desktop PCs
Network Devices
Routers, switches, hubs, repeaters, etc.
– If you’re lucky, IT has standardized hardware (e.g. all-Cisco, with multiple routers & switches from the same model families)
– Use the “cookie-cutter” “horizontal” approach: fully OQ one device from each family, then record the installation and configuration data for all devices in that family
– Verify what you’ve specified in your design document. Examples:
Networking protocols you’ve specified are configured
Connectivity, bandwidth, and (if important) latency across links
Redundancy, if you’ve built it in
The “Cookie Cutter” or “Horizontal” Qualification Approach
Role: Access SwitchMinimum RAM: xxxMinimum Flash: xxxSpanning tree protocolTACACS authenticationetc.
3. Many instances of the device are deployed
1. A configurationis specified for
a role:
2. One device is fully challengedSystem: Catalyst 65xx; software: IOS 12.x
Network Device Configuration Settings
Important: recording network device configuration includes the configuration settings done in software (e.g. in IOS)
This is critical, and it changes often (e.g. new static routes, access lists, packet filters)
You need to “version” these config settings
Print them out if you have to
Better still, copy each version into a secure directory or document management app
Even better: use a network management tool (e.g. CiscoWorks)
Network Services
Low-level services = little qualification work
– NTP, DNS, DHCP
– Don’t forget: broken low-level services can have high-level impact
More testing and documentation for higher-level record-critical or security-critical services
– Authentication services (e.g. Active Directory, LDAP, NIS+, )
– Remote access & VPN services
– Firewalls
– Virus protection
– Backup/restore solutions
– Network management software
Data Centers
Recommended approach:
– Unless your company is fat with cash, your data centers will look very different from site to site… so don’t set your standards arbitrarily high!
– Focus on compliance with building/safety codes, physical security, and meeting the requirements of the equipment within
– Leverage commissioning paperwork if possible to show that power, HVAC, fire, and security systems are installed per manufacturer specifications and have proper capacity
– Verify proper equipment loading (e.g. breakers, batteries, cooling)
– “Testing” is mostly a lot of looking, verifying, measuring, documenting
– Runs from subjective (“This rats nest under the floor is ugly and needs fixing”) to objective (“These cables are blocking air flow”)
Desktop PCs
Recommended approach:
– Qualify a “golden image” that you’ll burn onto each PC
– Qualify an approved standards list for hardware & software add-ons
– Limit the changes that users can make (e.g. no local admin privs; Active Directory controls, approved software add-on lists)
– Control the changes that IT can make (e.g. qualify electronic software distribution tools and changes to the golden image; use SOPs to control these activities)
Recommendation: don’t even bother with change control for your PCs unless they’re running GxP software locally(e.g. standalone or “fat client”)
Desktop PCs
Enforce standardization of hardware & software
Standardize on thin client applications (e.g. Citrix, Java, ActiveX)
– If your software configuration specifies only “IE10 or better under Windows 7 SPx or better”, your qualification job is pretty easy
Test the golden image with all GxP apps
If feasible, test all software on the “Approved Add-Ons” list with all GxP apps
Challenge any controls that are expected to keep the desktop configuration locked down
– Examples: users don’t have local admin privs; users can’t install applications; users can’t change the clock
The “Cookie Cutter” or “Horizontal” Qualification Approach
Baseline ConfigurationFamily: Lenovo xxxxBIOS: xxxMinimum RAM: xxxApproved devices…Windows settings…Installed apps…
3. Many instances of the family are deployed
1. A configurationis specified for
a series:
2. One machine running the “golden softwareimage” is qualified
Network PQ
Recommended!
– The protocol should be as simple as formally monitoring the network for a period of time (say 30 – 60 days)
– Record any incidents, problems, anomalies, capacity issues, etc.
– Document resolutions (or plans for resolutions)
Qualification Final Report
Summarizes the outcome of each protocol
– If each protocol had its own report, the final report just summarizes those
– If not, then the final report should explain/rationalize any deviations that were encountered along the way
– If any non-critical deviations are still “open” (e.g. were resolved by change controls), the final report lists these along with the change control numbers
Declares the infrastructure “qualified”
Maintaining The Qualified State
Necessary procedures and work instructions
– Change control & configuration management: to maintain that qualified state
– Operations: assure regular monitoring, backups, preventive maintenance, etc.
– Security: password rules, access rights by role, periodic audits
– Others (incidents & problems, capacity management, training, etc.)
Monitor the ongoing performance of your network, servers, and applications on an ongoing basis
Keep qualification documents, network diagrams, and configuration data up to date as the network changes over time
– Pointers to current and prior electronic versions are okay
Test new equipment/configurations as appropriate
Think about executing an overall network PQ on a periodic basis
Change Control and Infrastructure
After qualification, you’ll have many new GxP items to track –each with its own significant configuration challenges
– Supplier changes
– Router/switch configuration changes
– New or reconfigured building
– Desktop hardware refreshing
How can you put infrastructure under change control when it changes so often?
– Decide what change types need QA involvement
– Think ahead and develop a routine change list...
Routine Changes
A pre-defined list of infrastructure changes that can be implemented by IT without ‘formal’ change control
Pre-approved by QA; need no hands-on QA or CV involvement
Typically low-risk/low-impact activities
Examples:
– Adding a previously qualified model of network device
– Reboot server or restart application/service
– “Like-for-like” hardware changes (e.g. disk in RAID 5 array)
– Updating virus .dat files
– Windows security patches? Service pack updates?
Routine Changes
Caveats
– Be very clear and specific in your wording to prevent misinterpretation of “like-for-like”
– Some routine changes must be tested in a non-production environment
– All changes require some type of documentation (e.g. work request, system log)
– History of routine changes must be accessible
– Requires periodic auditing
Events/Problems/Incidents/Deviations
Similar forethought should be given to dealing with infrastructure incidents
Certainly, not every ITIL incident constitutes a GxP incident
The IT SOP or work instruction for handling incidents/problems should have logic that triggers a Quality SOP if appropriate
Possible examples:
– A problem with a GxP server
– A problem with barcode scanners used by a GxP application
– A network intrusion or virus infection
Maintaining The Qualified State
The Quality unit should “Trust…
– Many changes and events/incidents/problems should be handled by IT without direct Quality or Validation involvement
– IT procedures should be in place to guide the proper degree of planning, documentation, testing, verification, reporting, etc.
– The best compliance comes when IT is self-enforcing
– IT needs to understand when to involve Quality
… but Verify”
– Periodically, the independent Quality unit should audit IT operations and documentation to assure that procedures are being followed
Interactive Workshop
Questions?
