Embed Size (px)
DESCRIPTION
JLab Software Assurance Program. A Risk Based Approach to Software Management. Outline. Software Assurance vs. Software Quality Assurance QA Order Requirements Processes that address software assurance JLab’s SW Assurance Effort Risk Based Model Assessment Method Preliminary Results - PowerPoint PPT Presentation
Citation preview
JLab Software Assurance Program
A Risk Based Approach to Software Management
2
Outline
• Software Assurance vs. Software Quality Assurance• QA Order Requirements• Processes that address software assurance• JLab’s SW Assurance Effort• Risk Based Model• Assessment Method• Preliminary Results• Path Forward
8/18/2010
3
DOE 414.1C• Applies to ALL Software activities• Ten Criteria for Safety SW QA Program• Requirements flow-down through CRD
QA Order414.1C
QA PLAN
CORRECTIVE ACTION
MANAGEMENT
SUSPECT/COUNTERFEIT ITEM PROCESS
(SAFETY) SOFTWARE QUALITY
SOFTWARE ASSURANCEPROCEDURE
GENERALREQUIREMENTS
8/18/2010
4
JLab Software Assurance Procedure
• Implementation of Requirements of Quality Assurance Plan
• Implements a process for identifying and classifying the impact SW may have on multiple subject areas, including safety
• Adaptable to all software activities important to facility mission and goals
• Implements consistent tiered approach
8/18/2010
Software Quality Assurance Software AssuranceNASA-STD-8739.8 (w/Change 1) July 28, 2004“Software assurance consists of the following disciplines:• Software Quality
– Software Quality Assurance– Software Quality Control– Software Quality Engineering
• Software Safety• Software Reliability• Software Verification and Validation (V&V)• Independent Verification and Validation (IV&V)”
NASA-GB-8719.13 NASA Software Safety Guidebook. • Implementation guidance for high consequence software
8/18/2010 5
6
JLab Process• SW Assurance team chartered by CIO
– Representatives from across site:• Scientific
– Experimental– Theoretical– Computing
• Business• Controls• Cyber Security• HR• Safety Systems
8/18/2010
7
JLab SWAP - Table of Contents
1 Purpose1.1 Structured Approach
2 Scope2.1 Exemptions
3 Responsibilities4 Software Control Procedure4.1 Software Risk Assessment Process Steps & Expectations4.1.1 Software critical to JLab safety, operations, and mission
4.1.2 Software important to JLab safety, operations, and mission
4.1.3 Software Risk Assessment Assumptions:4.1.4 Software Risk Assessment Tool8/18/2010
8
Table of Contents - Continued4.2 Software Assurance
4.2.1 Graded Approach4.2.2 Software Assurance Program Requirements
4.2.2.1 Acquirer Software Assurance4.2.2.2 Basic Requirements for Software Assurance
Processes:4.2.2.2.1 Software Lifecycle4.2.2.2.2 Software Quality4.2.2.2.3 Competence4.2.2.2.4 Sustainability4.2.2.2.5 Configuration Management4.2.2.2.6 Assessment
4.3Metrics and Continuous Improvement5.0DEFINITIONS6.0REFERENCES7.0REVISION SUMMARY
8/18/2010
9
JLab SW Assurance Procedure • Clarifies scope:
– Applies to all projects, programs, facilities, and activities that may impact JLab mission and goals
– Applies to all software developed, or modified for use at JLab.
• Compliments Cyber Security Program– Reflects SW Enclaves– Applies to security software configuration items only
where ineffective security software controls may directly affect operations and safety
– Cyber Security Risk Assessment incorporated in to overall risk assessment
8/18/2010
10
JLab SW Assurance • Defines SW Risk Assessment Procedure:
– Identifies pre- and post-mitigation Risk– Applicable to ALL SW within scope of process
• Defines Requirements for Owner Organization SW – Requirements for lifecycle model– Requirements for
8/18/2010
11
Structured Approach
• Identify important JLab software configuration items and activities
• Identify roles and responsibilities for software control activities within the context of this procedure
• Perform a software risk assessment on applicable configuration items
• Apply a risk-based graded approach to software assurance activities
• Apply a value added continuous improvement process to the software assurance processes
8/18/2010
12
Exemptions
“This procedure does not apply to unmodified general purpose computing software, unmodified enterprise software, and general purpose desk-top software managed under the IT/CIO Division. Examples include office productivity software, public web pages, and LAN/WAN networking software.”
8/18/2010
13
Roles and Responsibilities
• Defines roles, responsibilities, and authority for – COO– CIO – ESH&Q AD– Division Management– Line Management– Software Owner– Oversight committees
8/18/2010
Scope
• internal software development • software used to collect and
manage data• startup and configuration
scripts• incorporation of open source
software • modified off the shelf (MOTS)
software used to design, analyze, or control safety or mission essential aspects of JLab operations
• commercial off the shelf (COTS) software used to design, analyze, or control safety or mission essential aspects of JLab operations
• programs and firmware for monitoring or control, including IOCs and PLCs
• modifiable embedded software and firmware including PICs and PC104 type SBCs
• programs and development software for field programmable integrated circuits such as Field Programmable Gate Arrays
• Other software as defined by the JLab Chief Information Officer
SW Risk Assessment• Software is scored (1-5) in each of six areas of impact
– Direct Risk of Financial Loss– Direct Risk of Loss of Tangible Equipment/Property– Direct Risk of Harm to People– Direct Risk of Harm to the Environment– Direct Risk of Loss of Continuity of
Operations/Organization/Mission– Direct Risk of Enforcement Action
• Scoring is reviewed at both the individual and cumulative level
Data Collection
SW Application/InformationExample Worst Credible
Error/Event/Incident Owner Type
Assigned JLab Cyber Security
Level Platform
Application, information, or SW configuration item
Briefly describe event and consequences of SW error
Organization that owns the performance/requirements
for the applicationMajor SW
application type HW/SW platform
running the application
Auger lost productivity (physics "farm") IT - SCI Java Low Linux
Scoring
Direct Risk of Financial Loss,
Includes Cost of Unplanned Labor
Direct Risk of Loss of Tangible
Equipment/MaterialDirect Risk of Harm
to PeopleDirect Risk of Harm to the Environment
Direct Risk of Loss of Continuity of Operations/
Organization Mission
Direct Risk of Enforcement Action
Number from 0 to 5 (See Table- Definitions)
Number from 0 to 5 (See Table- Definitions)
Number from 0 to 5 (See Table- Definitions)
Number from 0 to 5 (See Table- Definitions)
Number from 0 to 5 (See Table- Definitions)
Number from 0 to 5 (See Table- Definitions)
3 0 0 0 2 0
Analysis Recommendations
Total Pre Mitigation SW QA Mitigation(s) Post Mitigation
Risk Acceptance SW QA practices used for this application Risk Acceptance
5 Tolerable Testing prior to release; testing by users Acceptable
Types of Software Assessed
– Lattice QCD Modeling– Experiment Data
Management– MIS Accounting– MIS Procurement– Corrective Action
Tracking System– Facilities Work Order
System– Travel– -Timesheets
– Facilities Work Order System
– Accelerator Physics Model
– Machine Protection– Personnel Safety
System• PLC Firmware• PLC Program
– Radiation Instrumentation
– Beam Position Monitor
Risk Acceptance Criteria – Pre Mitigation
1 100%
20%
40%
60%
80%
100%
Normalized Dist. Of Pre-Mitigation Risk Bins
IntolerableUnacceptableTolerable
Score Total
Nor
mal
ized
SW Assurance as a Mitigation
• Each Owning Organization responsible for tailoring requirements in to their own SA program
• Procedure provides consensus standard references for generally accepted good practice
• Procedure provides references for incorporation in to organization’s individual process
Lifecycle Model Requirements
Metrics
• Recommended for Acceptable and Tolerable Risk• Required for Intolerable and Unacceptable Risk• Assessments go back to central repository for analysis• Allows comparison of mitigations vs. claimed
effectiveness• Refers to existing SW metric processes• Guidance refers to CMMI process
– Pilot project complete– In process of implementing procedure lab wide– Feedback (mostly) positive– Expanding Risk Assessment data
Status
Conclusions
• JLab is implementing a risk based software assurance process
• Consensus based procedure with buy-in from all SW enclaves
• Tools, e.g. risk assessment spreadsheet, are integrated in to the process
• Provides minimum requirements for SW lifecycle• Incorporates resources and guidance for application• Process incorporates metrics
