xLPR Version 2 - NRC: Home Page · xLPR Version 2.0 Kick-off Meeting David Rudland, RES/DE/CIB . The Legacy Hotel and Meeting Centre September 28-29, 2011 Rockville, MD . 09/28/2011

09/28/2011 vg 1

xLPR Version 2.0 Kick-off Meeting

David Rudland, RES/DE/CIB

The Legacy Hotel and Meeting Centre

September 28-29, 2011 Rockville, MD

09/28/2011

Welcome - POP • Purpose

– To discuss the xLPR Version 2.0 code management structure, the development and implementation of the xLPR QA plan and the scope for the xLPR Version 2.0 code

• Outcome – Common understanding of Version 2.0 objectives, roles,

responsibilities, work plans, and scope – Identify follow up actions, if any

• Process – Present QA structure, task group work plans, hold QA

training, and facilitate task group meetings – Address any comments or questions

vg2

09/28/2011

Agenda - Wednesday

vg3

Wednesday, September 28, 2011

8:00 am to 8:15am Arrival & Check in.

8:15 am to 8:30 am Welcome, Agenda and Introductions David Rudland, NRC

8:30 am to 9:00 am xLPR Pilot Study Results Craig Harrington, EPRI

• Overview of xLPR Pilot Study Results • Status of reports/meetings

9:00am – 10:00am

xLPR Version 2.0 • Regulatory Perspective – (15min) Robert Hardies, NRC • Industry Perspective – (15min) Craig Harrington, EPRI • Goals – (30min) David Rudland, NRC

10:00 – 10:15am Break

10:15 to 11:30 am xLPR QA Program

• QA Document requirements – (15min) Nancy Kyle, Theseus • Software Program Management Plan – (1 hour) David Rudland, NRC

11:30 am to 12:30pm Lunch

12:30pm to 2:30pm xLPR QA Program – Continued

• Software Quality Assurance Plan – (1 hour) Nancy Kyle, Theseus • QA Administration Group work plan – (1 hour) Hilda Klasky, ORNL

2:30pm to 4:30 pm xLPR Version 2.0 Scope Discussion • Models Group work plan – (2 hour) Marjorie Erickson, PEAI

4:30 pm to 4:45pm Wrap-up Discussion and Adjourn Meeting

09/28/2011

Agenda - Thursday

vg4

Thursday, September 29, 2011

8:15 am Arrival & Check in.

8:30 am to 10:30pm

xLPR Version 2.0 Scope Discussion – Continued

• Input group work plan – (1 hour) Gary Stevens, NRC • Computational Group work plan – (1 hour) Patrick Mattie, SNL

10:30 am to 10:45 am Break

10:45am-11:15am xLPR Code Review Discussion Craig Harrington, EPRI

11:15 am to 12:00pm xLPR Acceptance Group Discussion Robert Hardies, NRC

12:00 – 1:00 pm Lunch

1:00 – 3:00pm QA training

3:00pm-5:00pm Individual Group Meetings

Pilot Study Results & Report Status

Craig Harrington EPRI xLPR Version 2.0 Kick-off Meeting September 28 - 29, 2011

2 © 2011 Electric Power Research Institute, Inc. All rights reserved.

Pilot Study Results

• The project team demonstrated that it is feasible to develop a modular-based probabilistic fracture mechanics code within a cooperative agreement while properly accounting for the problem uncertainties.

• The project team demonstrated that the cooperative management structure was promising, but recommends slight restructuring.

• The GoldSim commercial software will be used for future xLPR versions.


Lessons Learned & Key Gaps (Final Report)

• Organizational Issues – primarily addressed in revised organization • Communication Issues – partially addressed in revised organization

– Direct & Indirect Group Communication – revised leader responsibilities & continued overlapping meeting participation

• Framework Issues – GoldSim selected – Inputs and Outputs – Ongoing planning – Uncertainty Classification and Analysis – Planning required – Improved Sampling Techniques – Development required – Data Storage and Handling – Ongoing Planning – Post Processing - Function of framework development details

• Models Issues – Expertise – Awareness issue, not a critical weakness – Modeling Scope – Key area requiring detailed planning, etc.

• Software QA & CM – Development progressing well


Path Forward – Major Milestones

• Project management restructuring – in place

• Version 2.0 QA program development – nearing completion

• Team Lead Planning Meeting – August 2011 (complete)

• Version 2.0 Model and capability discussions – underway – Focus first on SCC initiation

• Version 2.0 begin model development – Sept 2011

• Version 2.0 Framework implementation – April 2012

• Version 2.0 V&V – April 2013

• Version 2.0 release – End 2013


xLPR Team Lead Planning Meeting

• Similar to current meeting’s agenda • QA/CM Program

– Reviewed development status – Reviewed remaining action items & assignments

• Version 2.0 Scope – High-level scope previously set – Evaluated detailed recommendations captured in TG reports

• Urgency to meet V2.0 goals • Practicality to implement w/in schedule • Ease of incorporation

– TGs responsible to refine detailed scope for approval – Work Plans being drafted and will be presented in this meeting


Report Heirarchy

xxxLLLPPPRRR PPPiiilllooottt SSStttuuudddyyy FFFiiinnnaaalll RRReeepppooorrrttt

GGGSSSxxxLLLPPPRRR UUUssseeerrrsss MMMaaannnuuuaaalll

SSSIIIAAAMMMxxxLLLPPPRRR UUUssseeerrrsss MMMaaannnuuuaaalll

xxxLLLPPPRRR VVVeeerrrsssiiiooonnn 111...000 RRReeepppooorrrttt

VVVeeerrrsssiiiooonnn 111...000 CCCooommmpppaaarrriiisssooonnn RRReeepppooorrrttt

xxxLLLPPPRRR VVVeeerrrsssiiiooonnn 111...000 GGGooollldddsssiiimmm

xxxLLLPPPRRR VVVeeerrrsssiiiooonnn 111...000 SSSIIIAAAMMM FFFrrraaammmeeewwwooorrrkkk

xxxLLLPPPRRR VVVeeerrrsssiiiooonnn 111...000 MMMooodddeeelllsss///IIInnnpppuuutttsss

To include: •Description of framework development

•QA and CM •Pilot study problem and results

•Sensitivity analyses •Code assessment and comparison with other

•Lessons learned •Recommendations for further xLPR development

Written by Computational group

xxxLLLPPPRRR VVV111...000 CCCooommmpppaaarrriiisssooonnn tttooo WWWiiinnnPPPrrraaaiiissseee

Written by Models & Inputs group

Written by SIA

Written by SNL

Written by ORNL

Written by CNWRA

NUREG/EPRI Report


Report List Details

Report Title Developed By Identifier

xLPR Pilot Study Final Report (MRP-315) In Review NRC & EPRI NUREG #

EPRI PID 1022860

GSxLPR and SIAMxLPR Comparison Report [7] CNWRA (Southwest Research Institute) ML111510924

xLPR Version 1.0 Report, Technical Basis and Pilot Study Problem Results [5]

xLPR Computational Group ML110660292

xLPR Framework (GoldSim) Model User’s Guide [9] Sandia National Laboratory

ML110700017

SAND2010-7131

Structural Integrity Assessments Modular-Probabilistic Fracture Mechanics (SIAM-PFM): User’s Guide for xLPR [10]

Oak Ridge National Laboratory ML110700023

Development, Analysis, and Evaluation of a Commercial Software Framework for the Study of Extremely Low Probability of Rupture (xLPR) Events at Nuclear Power Plants [2]

Sandia National Laboratory

ML110700019

SAND 2010-8480

SIAM-xLPR Version 1.0 Framework Report [3] Oak Ridge National Laboratory ML110700026

Models and Inputs Selected for Use in the xLPR Pilot Study (MRP-302) [4] In Publishing

xLPR Models/Input Groups EPRI PID 1022528

xLPR Version 1.0 GoldSim and SIAM Benchmark Testing with Comparisons to WinPRAISE Calculations (MRP-306) [8] In Review EPRI MRP / SIA EPRI PID 1022731


Together…Shaping the Future of Electricity

X LPR Regulatory Perspective

Bob Hardies

Division of Engineering Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission

U.S. Nuclear Regulatory Commission

LBB

• 10CFR50 Appendix A GDC-4 – Exclude local dynamic effects of pipe ruptures

from design basis if pipe ruptures have extremely Low PRobability

• Excluding local dynamic effects eliminates need for whip restraints and impingement shields

• Conservative flaw tolerance analyses incorporated in SRP 3.6.3 to demonstrate leak-before-break

2


xLPR

• Calculate rupture frequencies for evaluating PWSCC while considering: – NDE – Mitigation – Operational characteristics

• Modularity to permit adaptation to other anticipated applications

3


User Need

• Develop project plan • Complete feasibility study • Develop a modular code for LBB

– Perform independent review • Develop acceptance criteria • Evaluate specific and generic LBB applications • Draft a regulatory guide

4


Intended Application

• Procedures for evaluating new LBB applications • Procedures for evaluating the effect of modifications

or operational changes on LBB • In the long term the code will be adapted for other

uses

5

09/28/2011


Version 2.0 Goals




Slide 1

09/28/2011

Pilot Study Goals

• Version 1.0 of xLPR had distinct goals – Demonstrate feasibility of approach – Demonstrate feasibility of management structure to meet

goals – Choose appropriate computational framework

• We believe we met the goals for Version 1.0

• Version 2.0 has markedly different goals

vg2

09/28/2011 vg 3

xLPR Goal - Reminder

• Tool will be – Comprehensive with respect to known challenges and

loadings – Vetted with respect to scientific adequacy of models and

inputs – Flexible to permit analysis of a variety of in service

situations – Adaptable – able to accommodate

• evolving / improving knowledge • new damage mechanisms

• Developed within QA structure

09/28/2011

xLPR Driver • NRR-2010-018 – Probabilistic Method for LBB

– Deliver a flexible, modular probabilistic fracture mechanics code for evaluation of PWSCC in dissimilar metal butt welds - End of 2013 • Include active degradation modes • Include inspection/mitigation/repair strategies • Correctly quantify, characterize, and propagate uncertainties

– Deliver technical basis and regulation guide for LBB - 2015

vg4

09/28/2011

Flexible, Adaptable Code

• A code that meets the current NRR need, but remains flexible and adaptable to meet future application needs is desirable

• Pipes, pressure vessels, steam generators are all cylindrical objects that may contain pre-service, or service-induced cracks

• While geometrically similar, models and inputs may be quite different vg5

09/28/2011

Other Possible Applications • In addition to LBB and evaluation of mitigation for

DM welds – Research tool for prioritization – TBS – 50.46a – Risk informed ISI – GSI191 – HELB – Effects of seismic loading on integrity – Easily adaptable to other applications

• CRDM ejection probabilities • RPV

• Must have acceptability criteria and technical basis for each applications

vg6

09/28/2011

Version 2.0 Constraints

• Time scale – Per NRR need, Version 2.0 needs to be complete by end of 2013

• Cost –The success of the project is highly dependent on the availability of funds and resources – NRC/RES budget based on priorities of NRR – EPRI budget based on industry priorities for PWR

materials-related research

• Expert staff availability

vg7

09/28/2011

Version 2.0 Scope

• The goals for Version 2.0 must meet the NRR requirements, but still fit into the budget and resource constraints

• The scope will include as many “other” options as the team feels can fit into these constraints – Other degradation mechanisms – Other materials – Etc.

vg8

09/28/2011

Version 2.0 Scope

• Team leaders and PIB have recommended scope for Version 2.0

• Group work plans will define scope that accounts for these constraints

• If required scope violates the constraints, discussions between the Code Development Team and NRR will have to occur

vg9

09/28/2011 vg 10

xLPR – NRC Intended Use • Version 1.0 – Pilot study – Surge nozzle DM weld

– To demonstrate feasibility – Determine appropriate probabilistic framework – Develop plan for future version

• Version 2.0 – Primary piping – Support LBB Regulation Guide development – Assess compliance with GDC-4 – Prioritize future research efforts

• Version 3.0 – Reactor coolant pressure boundary – Combine piping with reactor vessel, steam generator, etc. – Analyze probability of failure for all coolant pressure

boundary components

xLPR Version 2.0 Kickoff September 28, 2011

xLPR Program Document Approach

Presented by: Nancy M. Kyle Principal Consultant Theseus Professional Services, LLC “Results Through Service Excellence®” 706-830-3194 / [email protected]

mailto:[email protected]

xLPR Version 2.0 Kickoff September 28, 2011 Slide 2

Program Document Hierarchy

Program Basis Requirements

Software Project Management Plan

Software Quality Assurance Plan

Software Configuration Management Plan

Individual Work Plans

Release Requirements Design Implementation Test


Program Basis Requirements 10 CFR50 Appendix B

Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants

Regulatory Guide 1.28 Revision 4, June 2010 Quality Assurance Program Criteria (Design and Construction)

NUREG/R-0167, February 1993 Software Quality Assurance Program and Guidelines

10 CFR830 Subpart A Quality Assurance Requirements

DOE Order 414.1D Quality Assurance

ASME NQA-1-2008 & NQA-1a-2009 Addenda Quality Assurance Requirements for Nuclear Facility Applications

Part I Requirements 1, 2, 3, 4, 5, 6, 7, 11, 17, 18, Part II, Subpart 2.7


Program Document Alignment

Software Project Management Plan Uses basic structure of IEEE1058-1998 ,

IEEE Standard for Software Project Management Plans

Incorporates the applicable NQA-1 requirements not specific to software, but necessary to manage the project Organization Training Procurement Instructions, Document Control, Records Management Reviews and Audits



Software Quality Assurance Plan Uses basic structure of IEEE-730-1998 ,

IEEE Standard for Software Quality Assurance Plans

Incorporates the applicable NQA-1 requirements that are specific to developing software, including the phased activities of: Requirements Design Implementation Testing Release Retirement

Points out global processes that will be addressed in supporting documents: Configuration Management Problem Reporting and Corrective Action



Software Configuration Management Plan Uses basic structure of IEEE-828-2005 ,

IEEE Standard for Software Configuration Management Plans

Incorporates the basic requirements for Configuration Management from NQA-1 Requirement 3 and Subpart 2.7 Describes “HOW” we will implement the Configuration

Management Process Applies across all project areas.

Also includes the process for Problem Reporting and

Corrective Action as xLPR is being developed.


Program Document Alignment Individual Work Plans

Identify the goals, objectives, activities, and deliverables for each Project Group or sub-group Applying the requirements of the SPMP and the SQAP to

specific tasks

Includes resource loaded schedules (to the extent possible) Personnel Timelines

Address Risks that could affect the schedule and how they are managed

Template has been developed for minimum required elements May be expanded to suit the needs of each group

09/28/2011 vg 1






09/28/2011

Software PMP • Based on IEEE Std 1058-1998, IEEE Standard for

Software Project Management Plans

• The purpose of the SPMP is to detail the organization, schedule, budget, and process the team will use during the development life cycle of xLPR Version 2.0

• The SPMP is a living document and may evolve as the code evolves.

• The xLPR team is the intended audience for this document

vg2

09/28/2011

Outline • Overview – Purpose, scope, assumptions, constraints,

deliverables, schedule and budget

• Project organization – External interface, internal structure, roles and responsibilities

• Managerial process – Work plans and control plans

• Technical process – Software model and differing professional opinions

• Supporting processes – Training, document control, etc.

vg3

09/28/2011

Project Constraints

• Regulatory need for quantitatively assessing compliance with GDC-4 for Leak-Before-Break

• Code being developed cooperatively between NRC and EPRI though Addendum to Memorandum of understanding

• The success of the project is highly dependent on the availability of funds and resources – NRC/RES budget based on priorities of NRR – EPRI budget based on industry priorities for PWR

materials-related research vg4

09/28/2011

Schedule + Deliverables

vg5

09/28/2011

xLPR Organization

vg6

09/28/2011

xLPR Code Development Group • Development of modular-based probabilistic fracture

mechanics code capable of determining the probability of failure for Reactor Coolant System (RCS) components – Lead – David Rudland – Co-lead – Craig Harrington

• Coordinate with – Code review groups to ensure technical consistency – Stakeholders to ensure consensus between the NRC and

Industry on technical issues and code specifics – Acceptance group to ensure smooth progress of the

appropriate application specific technical basis – NRC/EPRI Program Managers to assure that the

appropriate contracts are in place vg7

09/28/2011

xLPR Code Development Groups

• QA Administration – Preparing and maintaining the QA documents – Coordinating, defining and controlling the program and

modular development revision levels. – Researching, acquiring, and implementing the proper

publically available configuration management system – Ensuring that all team members are properly following the

QA protocols

• Lead – Hilda Klasky • Co-Lead – Nancy Kyle

vg8

09/28/2011


• Computational – Integrate the computational elements into a robust, fully

developed, tested, and verified computational tool – Verify all models are coded appropriately, and are

adequately linked – Develop the overall modular structure including uncertainty

handling and appropriate sampling methods – Provide code documentation and training when necessary

• Lead – Patrick Mattie • Co-Lead – David Harris

vg9

09/28/2011


• Models – Selection, documentation, and coding of the mathematical

models for the prediction of probability of pipe rupture – Responsible for developing and using a comprehensive

ranking system for the selection of appropriate models to use in the xLPR code

– Aid in the quantifying of uncertainties

• Lead – Marjorie Erickson • Co-Lead – Matthew Kerr

vg10

09/28/2011


• Inputs – Identifying and collecting data and their associated

distributions to quantify the various input parameters – Aid in determining the best format for supplying input data

to the xLPR code, e.g., database – Aid in the quantifying of uncertainties

• Lead – Guy DeBoo • Co-Lead – Gary Stevens

vg11

09/28/2011

Code Review Group • Provide internal and external reviews of the code

development efforts – Program Integration Board (PIB)

• Reviews all aspects of development and makes recommendations

• Lead – Denny Weakland, Co-lead – Rob Tregoning – Advisory Committee on Reactor Safeguards (ACRS)

• Annual briefings expected • Lead – Michael Benson

– External Review Board (ERB) • Provide additional technical review for national and

internationally recognized experts that are not affiliated with the xLPR project

• Lead – Mark Kirk

vg12

09/28/2011

Acceptance Group • Develop the application-based technical basis for

– Results acceptability limits – Guidelines for using xLPR to obtain the application-specific

results. • Determine form of results to support use of the xLPR

evaluations as a basis for regulation and/or ASME code implementation

• Lead – Robert Hardies • Co-Lead – Robin Dyle

vg13

09/28/2011

Managerial Process • Work plans

– Each of the Code Development Groups are required to create an Individual Work Plan (IWP)

– Discusses work activities, schedule, budget and resource allocation.

– Aligned with the overall project goals, budget and schedule – Group work plans to be discussed

• Control plans

– SQAP – to be discussed – Reporting plan – Contractors per contracts - Meetings as

needed

vg14

09/28/2011

Managerial Process

• Control plans (cont) – Risk management plan – if availability of funds become

limited or inability to meet schedule occur, code development lead and PMs will meet with industry and regulators to discuss impact on schedule

– Close-out plan – Contractor reports and a final report will be generated

vg15

09/28/2011

Technical Process

• Process Model – – Spiral lifecycle model – Risk-driven approach – Addresses

requirements engineering through prototypes

– Addresses risk management by performing risk analysis

vg16

09/28/2011

Technical Process

• Resolution of Alternate Professional Views (APV) – Within group decisions to be made on majority basis

(Decision making process guide being developed) – Guidance document will be written to handle APV that will

allow • Task group to identify, record and resolve any appropriate

APV • A formal response to the APV is to be developed by the Task

group – QA Administration Group will develop guidance document

with input from Code Development groups

vg17

09/28/2011

Supporting Processes

• Training – Required on an as-needed basis

• Procurement Management – Code Development Lead to work with PMs to assure supplier meets procurement requirements

• Document Control – xLPR-SCMP

• Records Collection – SharePoint for now

• Project Reviews – Per Code Review Team

• Process Improvements – Per IWP revisions

vg18

09/28/2011

Approval

• The xLPR-SPMP document approval chain – Reviewers

• Code Review Lead, Acceptance Lead – Approver

• Code Developer Lead

vg19


xLPR Software Quality Assurance

Plan Presented by: Nancy M. Kyle Principal Consultant Theseus Professional Services, LLC “Results Through Service Excellence®” 706-830-3194 / [email protected]











Program Basis Requirements 10 CFR50 Appendix B

Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants

Regulatory Guide 1.28 Revision 4, June 2010 Quality Assurance Program Criteria (Design and Construction)

NUREG/R-0167, February 1993 Software Quality Assurance Program and Guidelines

10 CFR830 Subpart A Quality Assurance Requirements

DOE Order 414.1D Quality Assurance

ASME NQA-1-2008 & NQA-1a-2009 Addenda Quality Assurance Requirements for Nuclear Facility Applications

Part I Requirements 3, 11, Part II, Subpart 2.7


SQAP Structure Software Quality Assurance Plan Uses basic structure of IEEE-730-1998 ,

IEEE Standard for Software Quality Assurance Plans

Incorporates the applicable NQA-1 requirements that are specific to developing software, including the phased activities of: Requirements Design Implementation Testing Release Retirement

Points out global processes that will be addressed in supporting documents: Configuration Management Problem Reporting and Corrective Action


Structure Introductory Sections

Purpose Reference Documents Management (Roles and Responsibilities Documentation

Global Processes Software Reviews (performed throughout the life cycle Software Configuration Management Problem Reporting and Corrective Action

Software Life Cycle Methodology Planning Phase Requirements Phase Design Phase Implementation Phase Release Phase Maintenance Phase Retirement Phase

Supporting Processes Software Acquisition Interim Use of Unqualified Software Standards, Conventions, and Other Work Practices Tools and Techniques Supplier Control Risk Management Glossary


SQAP Introductory Sections Purpose Identify the QA requirements for the development,

procurement, testing, and configuration control of xLPR Module and Framework software for the Development Phase (Version 2.0 ) of the xLPR Project

Reference Documents Identifies documents used as a basis or reference for the

SQAP Management (Roles and Responsibilities) Roles specifically associated with SQA Functions

Project-level Roles and Responsibilities defined in the SPMP Additional granularity specific to SQAP scope

Introduction of roles required for execution Documentation Documents produced while implementing the SQAP


SQAP Global Processes Sections that Span the Life Cycle of the Project

Software Reviews Performed throughout life cycle to verify compliance to QA

requirements Internal to the development process Align with key project milestones

Software Configuration Management Spans entire life cycle to address:

Configuration Identification Change Control Status Control

Process addressed in Software Configuration Management Plan Problem Reporting and Corrective Action

Informal process for xLPR Version 2.0 Developmental phase with no user community

Process addressed in Software Configuration Management Plan


Software Lifecycle Methodology

Depicts layers of refinement from inception through elaboration, construction, transition and release

Iterative approach allows increased understanding of the problem through successive refinements.

Each iteration ends with an executable release.

Spiral Model


Software Life Cycle Methodology

Spiral Model – An Alternative View

Planning through Testing/ Assessment, for the four development phases Inception Elaboration Construction Transition (Release)

The process can be described in two dimensions, or along two axis: Horizontal Axis: represents time and shows

the dynamic aspect of the process as it is enacted, and it is expressed in terms of cycles, phases, iterations, and milestones.

Vertical Axis: represents the static aspect of the process: how it is described in terms of activities, artifacts, and workflows.


Software Life Cycle Phases Phases focus on activities to be performed and the resulting deliverables

Planning Phase Define the strategy and framework for the execution of the

xLPR Project Deliverables:

Software Project Management Plan (SPMP) Software Quality Assurance Plan (SQAP)

Establish the roadmap for all activities required to execute the project

Address all requirements – the “whats” Identify next level of documents necessary to provide

implementing details – the “hows” Individual Work Plans (IWP)

Slide 11

Software Life Cycle Phases

Requirements Phase Define high level requirements and acceptance criteria

and methods for each module Deliverables:

Software Requirements Description (SRD) Using IEEE-830 as a template

Requirements Document Criteria Form xLPR-SQA-3

Software Verification and Validation Plan (SVVP) Using IEEE-1012 as a template

Verification and Validation Plan Criteria Form xLPR-SQA-4


Slide 12


Design Phase Design activities carried out to the level of detail

necessary to perform verification that ensures design meets requirements.

Software Design Description (SDD) Using IEEE-1016 as a template

Design Document Criteria Form xLPR-SQA-5 Software Design Review


Slide 13


Implementation Phase Design is translated into code using coding standards,

conventions, and design specifications and reviewed for adherence to those standards, conventions, and specs.

Deliverables: Implementation Documentation

(Not a document, but a collection of deliverables) Computer program listings Source code Executables User Manual

Implementation Documentation Criteria Form xLPR-SQA-6 User Manual

SQAP identifies specific NQA-1 requirements, May use IEEE-1063 as a template

User Manual Criteria Form xLPR-SQA-7


Slide 14

Software Life Cycle Phases Test Phase Process of exercising or evaluating work products through

peer reviews and tests to demonstrate that software: Adequately and correctly performs all intended functions Properly handles abnormal conditions and events Does not perform adverse unintended functions

Deliverables: Test Plans Test Cases

Test Plans and Test Cases use IEEE-829 as a template Test Scripts Software Test Plan Criteria Form xLPR-SQA-8 Verification and Validation Report

Use IEEE-1012 as a template Software Verification and Validation Report Criteria Form xLPR-

SQA-9


Slide 15

Software Quality Assurance Plan Release Phase Maintenance Phase Retirement Phase Since xLPR Version 2 focuses on the development of the

software prior to release, detailed requirements of these phases will be developed at a later time. Sections currently have basic information that would not be

adequate in an operational environment but identifies the intent of what will be implemented and the requirements that must be satisfied.

Expectation that release is under an NRC process


Slide 16

SQAP Supporting Processes Software Acquisition All acquired software is considered as “Otherwise Acquired”

Not developed under a program approved consistent with NQA-1 Code team shall develop a Software Dedication Plan

Evaluation of acquired software Identifies activities to be performed and documentation necessary to

accept software for its intended use and place it under configuration control.

No acquired software is considered to be safety software Does not require formal commercial grade dedication

Interim Use of Unqualified Software Software required to support interim analyses used prior to

full qualification. Programmatic Decision (PD) Software.

Used only for scoping, planning, or driving PDs xLPR Version 2.0 Kickoff September 28, 2011

Slide 17

SQAP Supporting Processes Standards, Conventions, and Other Work Practices Identifies the use of IEEE Standards to meet the requirements

of NQA-1-2008 and 2009 addenda References existing programming practices document

developed for the xLPR Pilot Project Recommended Programming Practices for the xLPR Pilot Project

Tools and Techniques Describes use of custom-developed and commercial software

Fortran, C, C++, Pascal GoldSim Simulation Software

Describes general compatibility requirements


Slide 18

SQAP Supporting Processes Supplier Control Required for acquisitions where there is an established

supplier relationship. GoldSim is only identified product

Risk Management Approach addressed in SPMP

Will be a required element of Individual Work Plans for specific modules

Glossary Based on IEEE Std. 610-12-1990 Intended to identify and define unique terms used in the SQAP


Presented by:

Hilda Klasky

PISA - ORNL xLPR Version 2.0 Launch Meeting September 28-29, 2011 Legacy Hotel Rockville, MD

xLPR Version 2.0 Software Configuration Management Plan

2 Managed by UT-Battelle for the Department of Energy

Acknowledgments

• I would like to acknowledge contributions to this presentation from:

• Richard Bass, Nancy Kyle, Dave Rudland, Paul Williams.


Table of Contents

This presentation will cover the following topics:

• A glossary of terms

• Objectives of the Software Configuration Management Plan (SCMP)

• Overview of major divisions and sections of the SCMP

• Brief description of each section of SCMP

• Summary and Conclusions


Glossary: Configuration Management (CM) is the discipline of applying technical and administrative direction and surveillance to documents/software in order to

• identify and document the functional and physical characteristics of a configuration item,

• control changes to those characteristics,

• record and report change processing and implementation status, and

• verify compliance with specified requirements


Glossary: Configuration (Change) Control is an essential element of configuration management

• Includes evaluation, coordination, approval or disapproval of changes to configurations items

• implementation of changes to those configuration items after formal establishment of their configuration identification.

Configuration Item (CI) is an aggregation of hardware, software, or both

• that is designated for configuration management and

• treated as a single entity in the configuration management process.


Glossary: Change Control Board (CCB) consists of the Group Lead, Technical Reviewer, and the SCM Coordinator that

• approves the release of CIs,

• provides change control prior to procurement and/or development of software,

• pre-approves the use and implementation of a peer review process for software validation, and

• reviews and approves software QA documentation

• can be expanded as necessary to cover the workload and supply needed expertise


Glossary:

Baseline Software is an item or product which

• has been formally reviewed and agreed upon,

• serves as the basis for further development, and that

• can be changed only through formal change control.

Revision Control is the process of managing multiple versions of a piece of information.


Glossary:

Software Problem Reporting (SPR) Process is a process of identifying, reporting, and evaluating errors in software to ensure that

• software problems are identified and documented,

• all affected parties are notified, and

• all affected work is identified, evaluated, and revised as necessary.

Version Tracking System is a tool that manages changes performed on CIs.


Objectives of the Software Configuration Management Plan (SCMP) include

• Documenting and informing project stakeholders about SCM within the project,

• Identifying what SCM tools and practices will be used,

• Describing how the SCM tools and practices will be applied by the project’s team to promote success,

• Ensuring completeness, consistency, and correctness of CIs, and

• Controlling storage, handling, and delivery of the CIs


SCMP structure incorporates elements of IEEE and NQA-1

• uses basic structure of IEEE-828-1998 – Standard for Software Configuration Management Plans

• includes applicable NQA-1 requirements specific to identifying and tracking changes on CIs such as – Change control – Status control – Media Control


This overview of the SCMP structure will focus on three major divisions

• Introductory sections

• Software Configuration Management Activities

• Supporting Sections


SCMP Introductory Sections address five topics

• Purpose: briefly explains why the Plan exists and who the intended audience is.

• Scope: address applicability, limitations and assumptions on which the Plan is based.

• Reference Documents: Source documents mentioned while writing the SCMP.

• Management (Roles and Responsibilities): describes allocation of responsibilities and authorities for SCM activities to organizations and individuals with in the project.

• Documentation: Documents produced while implementing the SCMP.


Software Configuration Management Activities contains the core sections of the document

• Configuration Identification

• Configuration Control

• Change Control

• Status Control

• Media Control

• Evaluations and Review

• Release Management and Delivery

• Issue Management


Software Configuration Management: Configuration Identification • Lists the CIs

• Describes how each CI is to be uniquely named.

• Identifies artifacts not maintained as CIs


Software Configuration Management: Configuration Control

• Describes how to define baselines

• Describes the activities performed to track, store and retrieve CIs through the baseline managing process.


Software Configuration Management Change Control

• provides an organized and formal method for managing change requests to baselined CIs.

• defines the following sequence of specific steps: – Identification and documentation of the need for a change;

• a) The name(s) and version(s) of the CIs where the problem appears;

• b) Originators name and organization; • c) Date of request; • d) Indication of urgency; • e) The need for the change; • f) Description of the requested change.

– Analysis and evaluation of a change request – Approval or disapproval of a request – Verification, implementation, and release of a change.


Software Configuration Management Status Control • describes the activities to perform in order to

record and report the status of the CIs

Media Control

• identifies the media to control the CIs.


Software Configuration Management Evaluations and Reviews

• describes configuration audits as management tools for establishing a baseline.

• a) Its objective;

• b) The CIs under audit or review;

• c) The schedule of audit or review tasks;

• d) The procedures for conducting the audit or review;

• e) The participants by job title;

• f) Documentation required to be available for review or to support the audit or review;

• g) The procedure for recording any deficiencies and reporting corrective actions;

• h) The approval criteria and the specific action(s) to occur upon approval.


Software Configuration Management Release Management and Delivery

describes the following: • process to reconstruct any release. • communication process to announce new

releases. • location to store new releases. • information associated with new releases:

o Release notes. o Defects resolved. o New features added.


Software Configuration Management Issue Reporting - Problem Reporting and Corrective Action • describes process to report issues associated with

CIs as they are discovered while they are being used.

• issues can arise due to a change of requirements specification or due to implementation of CI not conforming to

requirements specification.

• Corrective Action section describes how issues discovered will be corrected to ensure issues in CIs have been addressed.


SCMP Structure Includes Several Supporting Sections

• Resources

• Schedules

• Maintenance

• Glossary

• Attachments (Forms)

• Annexes


SCMP Supporting Sections Resources • identifies software tools, techniques,

equipment, personal and training necessary for the implementation of the SCM activities.

Schedules

• describes when the SCM activities are going to be performed.


SCMP Supporting Sections Maintenance

Describes

• Who is responsible for monitoring the SCMP,

• How frequently updates are to be performed,

• How changes to the SCMP are to be evaluated and approved,

• How changes to the SCMP are to be made and communicated.


SCMP Supporting Sections Glossary • based on IEEE-610.12-1990

• intended to identify and define unique terms used in the SCMP.

Attachments • forms used through SCM activities.

Annexes • detailed information that may change as the project

develops.


Summary and Conclusions

• This presentation has focused on the description of an outline of the SCMP.

• It described • a glossary terms used In the report • the three major divisions of the report and • and the intent of each section within those

divisions.


Thank you for your attention!

Questions?

xLPR QA Administration Work Plan

Nancy M. Kyle QA Administration Co-Lead Principal Consultant Theseus Professional Services, LLC 706-830-3194 / [email protected]

Hilda B. Klasky QA Administration Lead PISA Oak Ridge National Laboratory 865-574-7602 / [email protected]




Work Plan Structure Description of the Group

Goals and Rationale for the Group

Group Objectives and Strategies for Achieving Them Potential Obstacles, Challenges and Other Risks to

Achieving Objectives Implementation Plan

Provides the Schedule and Resource Assignments for Each Objective

Planned Collaboration with Others Planned Group Travel Metrics and Reporting


Group Goals and Rationale

This is really what we do…


Group Goals and Rationale As defined in the xLPR Project Management Plan, the QA Administration

Role is as follows:

This task group shall prepare and maintain the QA Planning documents and all supporting implementing documents. The group is responsible for coordinating, defining and controlling the program and modular development revision levels. The group is responsible for researching, acquiring, and implementing the proper publically available configuration management system as well as for oversight of the QA program and ensuring that all team members are properly following the QA protocols.

The QA Administration Group provides Quality Assurance support for the xLPR Project. The QA Administration Group works independently of all development activities in order to provide objective oversight to ensure activities comply with ASME NQA-1-2008 and ASME NQA-1-2009 Addenda and conform to the subordinate xLPR Program Documentation that has been developed to govern this project.

Group Co-Leads function as the liaison to the QA Administration Group for their work groups


Group Objectives and Strategies for Achieving Them

Program Development

Quality Assurance Resource

Configuration Management

Verification and Validation

Quality Assurance Training

Quality Assurance Audits and Reviews


Group Objectives and Strategies for Achieving Them Program Development

Develop or facilitate development of program-level documents: Software Project Management Plan (developed by Code Development Lead (with input

from QA Admin) Software Quality Assurance Plan (developed by QA Administration) Software Configuration Management Plan (developed by QA Administration) Differing Professional Opinion Resolution Guidance Document (developed by Code

Development Lead (with input from QA Administration)

Develop templates to support most deliverables. These templates will be available to all project team members on the SharePoint QA Site.

Review each functional group’s compliance as they follow the processes defined in, and develop deliverables that are responsive to these program-level documents including Requirements, Design, Implementation, and Verification & Validation

Function as the “gatekeeper” for all of the program documents and deliverables for Version 2.0 of the project.


Group Objectives and Strategies for Achieving Them

Quality Assurance Resource Act as a resource to be utilized during the Leaders of each

functional group to ensure processes are being followed prior to completion of their deliverables.

Perform review of implementation strategies defined in each group work plan and identify tasks and deliverables that will require QA involvement.

Available to all project participants for guidance, and assistance with implementation.


Group Objectives and Strategies for Achieving Them Configuration Management

Responsible for the Configuration Management System for all configuration items identified by the project (e.g. documents, code, platform, metadata) and the problem reporting and corrective actions associated with these configuration items.

The QA Administration Group will: Document and inform project stakeholders about SCM within the project. Identify and implement the SCM practices and automated tools that will be

used. Describe how the SCM practices and tools will be applied by the project to

promote success. Manage the tools and their use throughout the life cycle Review and apply constant improvement to the SCM process as needed

throughout the life cycle.


Group Objectives and Strategies for Achieving Them Verification and Validation

The QA Administration Group will participate in verification and validation activities specified in Software Verification and Validation Plans (SVVP) generated by the Code Development Groups for each deliverable.

The QA Administration Group will verify that the Requirements Traceability Matrix is maintained for each deliverable and that the SVVP reflects the verification of this matrix.

These reviews will ensure that development phases flow smoothly through the project


Group Objectives and Strategies for Achieving Them Quality Assurance Training

Provide training to assure the project team understands the expectations of the program, and understands how to utilize the automated tools available (e.g. SharePoint, Subversion, JIRA)

Training will typically be included as part of a larger group meeting or other organized group meetings.

Refresher training will be provided annually and training information and resources will be available to new project participants when necessary.


Group Objectives and Strategies for Achieving Them Quality Assurance Audits and Reviews

Ensure the project fulfills the quality assurance requirements of NQA1-2008 (including Addenda 2009) that are described in the Software Project Management Plan and Software Quality Assurance Plan

Conduct Functional Configuration Audits to ensure that baseline configuration items are consistent with their documented specifications

Coordinate QA reviews of project milestone deliverables to assess quality assurance controls for each milestone defined in program documents and individual work plans. These reviews will assure that work practices are followed correctly, and that

processes are still aligned with these work practices.

Review system requirements contained in any Requests for Proposal for acquired software.


Obstacles, Challenges, and Risks Ability to remain aligned with all activities of each project group.

The group relies on a matrix structure for achieving the majority of goals and objectives.

Since the xLPR project team is dispersed through the country, it is affected by communication, network availability, and general interaction. The QA Administration Group will proactively engage groups, when necessary to

minimize the impacts on program implementation.

A QA function always faces the risk that the project team does not follow the program plans and implementation processes that are in place. QA Administration Group reports directly to the Code Development Lead with focus

on mitigating this risk through the objectives and goals of this work plan. The programmatic requirements of this project were developed on sound industry

expectations and implementation practices. Requirements will be communicated through training of the xLPR project team.


Obstacles, Challenges, and Risks

Reliance on external individuals for the configuration, system administration, and system support of the automated SCM tools used throughout the project. Risk of responsiveness from those external individuals when issues with these tools

arise.

The shared configuration management environment (Subversion) that the project uses is an Open Research environment with no information security controls. All information stored in this environment is recognized as public. All project team members need to be aware of this fact in order to avoid the

inclusion of any proprietary information.


Planned Collaboration with Others The scope of the QA Administration Group is to collaborate with all

work groups to facilitate the execution of their work scope. Review Work Group Plans

The QA Administration Group will review the Work Plans of each group in order to identify milestones and activities where QA involvement is needed.

Develop Execution Plans The QA Administration Group and Computational Group leadership will work together to

develop execution plans to assure these milestones are properly met.

Develop Deliverables The QA Administration Group and Work Group leadership will work together to develop

project deliverables that include the necessary and appropriate QA elements

Coordinate QA Reviews Coordinate QA review activities associated with the xLPR project.

Interface with External Service Providers Interact with the system administrators and support staff for the automated

tools used to support the project. SharePoint, Subversion, and JIRA.


Metrics Metrics will be developed to provide an indicator of the

progress relative to the objectives of the group. Metric: Documents and Forms Implementation

Measures the time it takes for xLPR users to complete documents through the baselining process

Identifies those forms and documents that take longer to complete and also which forms have more errors when being filled in

Provides feedback to the xLPR team to identify bottlenecks and needed improvements to the QA documents and forms

Metric: Number of source lines of code (SLOC): Used to measure the size of a software program by counting the

number of lines in the text of the program's source code Used to predict the level of effort required to develop a program, as

well as estimate programming productivity or maintainability once the software is produced.


Metrics Metric: Bugs per Source Lines of Code

Serves as a reference for further developments and as an indicator of how the ratio of bugs/SLOC behaves through the entire software development cycle.

Metric: Bugs per Module Identifies which modules did not comply with the requirements

specification after thorough testing. Offers a base of knowledge for future experiences with testing and

validation of requirements.

Metric: Program Execution Time Compares the xLPR v2.0 framework execution time, per realization,

against the time against the time to run the xLPR v1.0 framework on the same problem and the same computer.

Helps track performance improvement on the framework element of the xLPR final product.


Metrics Metric: Program Load Time

Measures the program load time (i.e., the time it takes to start the xLPR GoldSim framework, starting at the double click of the icon and ending at the time the program is fully loaded into memory and ready for execution)

We expect this time not to be significantly greater than the time it took on any previous major release (i.e. xLPR v1.0, xLPR v2.0 alpha, xLPR v2.0 beta and xLPR 2.0 production releases).

Metric: Binary File Size Measures the size of each GoldSim binary file and tracks its size

through the alpha, beta and production releases We expect file size to naturally grow and have no expectations to

reduce the file size, unless there is a performance issue.


Questions?





9/28/11

xLPR Kick-off Meeting Rockville, MD

Sept 28 & 29, 2011

Model Group Responsibilities • This task group is responsible for the selection, documentation,

and coding of the mathematical model building blocks for the xLPR code.

• This group is responsible for identifying or developing the appropriate models needed for the prediction of probability of pipe rupture including: – load models, – flaw initiation & growth models – stability models – mitigation models and – inspection models.

• This group is also responsible for developing and using a comprehensive ranking system for selection of appropriate models

• This group shall work with the Inputs and Computational group in quantifying uncertainties.

9/28/11

Models Group Work Plan

Models Group Team Structure Version 1.0

• Load/WRS • K-Solutions • Crack Growth • Crack Coalescence • Crack Initiation • Crack Stability (SC,TWC) • COD • Leak Rate • ISI • Mitigation

Proposed for Version 2.0

• Loads/Material Properties/ Mitigation (jointly with Inputs Group)

• WRS/K-solutions • Crack Growth • Crack initiation • Crack Stability • Leak Rate/COD • ISI


The Work Plan

• Is currently under development with a first draft available next week

• Requires input from each team following the work plan template posted on the sharepoint site

• By end of this meeting we need to have teams identified, and team scopes defined

• Scope details up next, followed by proposed milestones

9/28/11


Milestones & Timeline

9/28/11


Task Description QTR 3 QTR 4 QTR 1 QTR 2 QTR 3 QTR 4 QTR 1 QTR 2 QTR 3 QTR 41 Team Formation

1.1 Team structure finalization1.2 Team scope definition1.3 Team work plans complete2 Model Technical Development

2.1 New models selected2.2 Models revised

2.3Parameter uncertainties characterized & quantified

2.4 Models validated/benchmarked2.5 Software Requirements Description3 Module Development

3.1 Models Coded3.2 Module verified

3.3Modules Implemented into Framework

3.3 Software Design Description4 Module Implementation V&V5 Final Report

2011 2012 2013Models Group V2.0 Tasks

Models Group Work Plan 9/28/11

xLPR Version 2 Inputs Group Work Plan

Gary Stevens NRC RES

Guy DeBoo Exelon Nuclear

Presented at: xLPR Version 2 Kick-off Meeting Legacy Hotel and Meeting Center September 28-29, 2011 Rockville, MD

Objective

• To present the Inputs Group Work Plan for xLPR Version 2.0

Slide 2 xLPR Version 2 Kick-off Meeting - September 28-29, 2011

Background

• A straw man for the Inputs Group initial plan/scope for xLPR Phase 2 was presented at the Team Leaders Meeting on 08/31/11 – Feedback and input obtained from the Team Leaders to further

define/finalize the Inputs Group plan/scope

• Inputs Group Work Plan has been drafted

• This presentation reflects the current plan


Inputs Work Plan Scope • Each model requires inputs – some are calculated in other models – the

Inputs Group will only define those needed as external inputs to the xLPR Program, including uncertainties

• Need to focus on only those inputs needed for V&V efforts • Investigate all necessary inputs for:

– PWRs: • Westinghouse 4-loop plant

– Reactor Coolant Loop & branch lines > 6inch nominal, includes Alloy 82/182 welds and stainless steel welds

– Includes wrought and cast materials • B&W plant, reactor coolant system

– Hot leg and cold leg piping, includes Alloy 82/182, stainless steel, and ferritic steel welds

• CE plant loop piping – BWR:

• Recirculation, MS and FW piping systems, includes Alloy 82/182 welds, stainless steel welds and ferritic steel welds

• Only collect inputs to evaluate 2 PWRs (4-loop and B&W) with xLPR Code


Typical Westinghouse NSSS


Typical CE NSSS


Typical B&W NSSS


Typical BWR NSSS


Required Inputs

• Describe the number of, and differences between, the welds in each piping system: – Pipe, nozzle and weld geometry – Materials and material properties (strength, toughness, etc) – Fabrication processes – Documented weld repairs – Mitigation (type and time applied) – Operating loads/stresses/temperature – Operating transients (including earthquake) – Inspection parameters, schedule and results – Other?


Inputs Group Needs

• Other Inputs Group members: – Westinghouse contact for Westinghouse and CE plant designs – AREVA contact for a B&W plant – GE or BWRVIP contact for a BWR plant – Others? Uncertainty specialist?

• Where do we obtain the input flaw distribution?

• Make sure allowance is made in xLPR to accommodate BWR issues, i.e.:

– IGSCC growth and initiation models for stainless steel and Alloy 182 – Mitigation techniques:

• Induction Heating Stress Improvement (IHSI) • Mechanical Stress Improvement (MSIP) • Hydrogen water chemistry and noble metal

– Ability to address High Energy Line Break (HELB) requirements Slide 10 xLPR Version 2 Kick-off Meeting - September 28-29, 2011

Other Issues

• Issues that will affect what the Inputs Group can accomplish: – Proprietary data – Lack of data – Budget limitations – Lack of personnel

• Input data to be managed by database with Computational Group support

• Evaluation scope:

– Scope limited to just piping welds in primary (Class 1) RCS piping – Class 1 boundary defines piping system boundaries for evaluation – Piping diameters ≥ 4” NPS


Other Issues (cont’d)

• Are we addressing fatigue? If so, how? Do we need to define thermal transients? – The resources to obtain this information will be very demanding

• At least one Inputs Group member to attend all Models Group meetings

(and vice versa)

• Initial Inputs Group Meetings: – Telecon by end of October – Face-to-face meeting in January

• Acceptance Group will need to address acceptance criteria for an evaluation

• How do all welds in a system “stack” together? • How do multiple piping systems stack together and contribute to overall risk in the

plant? • In other words, “What is the risk limit (acceptance criteria) per weld? Per piping

system?” etc.


1

September 29, 2011

Presented by: Patrick D. Mattie, SNL

Computational Group Lead

Contributor: David Harris, SIA

Computational Group Co-Lead

xLPR Version 2.0 Computational Group Work Plan Briefing

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security

Administration under contract DE-AC04-94AL85000

xLPR Version 2.0 Kickoff Meeting – September 29, 2011– Legacy Hotel, Rockville, MD SAND#2011-6766P

2

Core Partners U.S. Nuclear Regulatory

Commission Sandia National Laboratories Oak Ridge National

Laboratories Pacific Northwest National

Laboratories Battelle EPRI Westinghouse Engineering Mechanics

Corporation of Columbus Structural Integrity

Associates, Inc. others

The xLPR Computational Group Team


3

Outline

Work Planning and Initial QA Documents

• Evaluation of Version 1.0– Pilot Study Results • Recommendations for Version 2.0 • QA Workshop and Work Planning

Computational Group Work Plan Activities

Schedule


4

Version 2.0 – Computational Group Work Plan

Computational Group Work Planning for Version 2.0 was based on the following process:

• Review of Version 1.0 Pilot Study Reports

Key lessons Learned

Recommendations for Version 2.0

• Review of SPMP and SQAP Requirements

Development of work plan for the computational group incorporated recommendations form internal and external review of xLPR pilot study model

Work plan incorporated both framework specific and recommendations for xLPR code computational improvements


5

xLPR Version 1.0 - Reports

SANDIA Report SAND2010-8480 “Development, Analysis, and Evaluation of a Commercial Software Framework for the Study of Extremely Low Probability of Rupture (xLPR) Events at Nuclear Power Plants”, December 2010.

ORNL/NRC/LTR-248 “SIAM-xLPR Version 1.0 Framework Report”, September 2010.

Center for Nuclear Waste Regulatory Analyses, “Assessment of Capabilities of Extremely Low Probability of Rupture (xLPR) Software – GoldSim and SIAM Version 1.0”, 2011.

Structural Integrity Associates, “xLPR Version 1.0 GoldSim and SIAM Benchmark Testing with Comparisons to WinPRAISE Calculations”, EPRI-xxxx, 2011.

NRC/EPRI “xLPR Version 1.0 Report, Technical Basis and Pilot Study Problem Results”, February 2011, ML110660292.


6

Key Lessons Learned from the xLPR Pilot Study

Costly re-work and schedule delays will result if the development process does not follow a QA process including work planning

Program goals must be clearly defined for and within each xLPR Group

Framework development must be integrated with module development efforts to ensure modularity and validity of information flow between the modules

Modules must be coded to a robust and consistent structure to avoid compilation errors and ad-hoc coding behavior that can delay the QA process (Code V&V)

Brute force Monte Carlo sampling was computationally expensive and should be improved in version 2.0 (long run times, large data sets, etc.)

Uncertainty quantification is not trivial and needs to be evaluated at all levels of the development process - Classification of uncertainty for key input can drastically change the distribution of the results (epistemic or aleatory)


7

Version 1.0 Evaluation and Recommendations

Input/Output • GoldSim Framework used Excel file to manage inputs. This has

limitations. • A simpler interface to select what data to save is desired. • Large output files for post processing that took a long time to export. • Many output files were generated to capture all of the necessary data

which was cumbersome, alternatives need to be considered

Classification of Uncertainty • Separation of epistemic and aleatory decreases model clarity and

increases runtime, needs to be revisited and optimized • Clear guidance on the consistent characterization of uncertainty for

inputs, models, and framework development


8

Version 1.0 Evaluation and Recommendations(2)

Due to the very low probability ruptures, some form of importance sampling must be included to get valid results without the long run times required for simple Monte Carlo approach

Post Processing • Simpler interface and usability of the post processing tools is needed. • Additional tools/methodologies need to be explored to enhance

parameter sensitivity analyses • Include tools/methodologies to investigate non-monotonic relationships

Framework software specific • CNWRA recommended investigation of GoldSim software features to

optimize code and decrease complexity • Eliminate or reduce redundancy in parameters required with the double

loop approach including the use of submodels • Improvements to the crack arrays to reduce complexity • Software vendor improvements to improve post processing/data

analysis


9

Version 1.0 - Evaluation Summary

While many advantages were identified in the selected framework approach for xLPR, the evaluations of the pilot study model suggested several areas for potential improvement.

Recommendations were compiled across a diverse spectrum of internal program members and external peer review teams including NRC, Industry (EPRI), Sandia, CNWRA, and other xLPR team members.

Recommendations for the CG focused on two key areas: • Improving the usability (user interface, understanding model structure

and results, etc.) • Improving the accuracy and computational efficiency

Independent reviews confirmed internal recommendations and key

lessons learned


10

Version 2.0 Work Planning and QA

QA Workshop and Expert Panel – March 31, 2010 xLPR Software Project Management Plan (SPMP)

• Organizational Structure • Overall Schedule and Scope for Version 2.0 xLPR Code • SQAP • Work Planning and Controls

xLPR Software Quality Assurance Plan (SQAP) • Configuration Management Plan • Checklists and Guidelines

Computational Group Work Plan – xLPR-WP-CG1 • https://websps1.battelle.org/nrcnureg/home/QA

NRCNUREG > xLPR_QA > Shared Documents > Computational Group Implementing Documents

• Outlines tasks for version 2.0 code development • Iterative process, update to work plan based on outcome of tasks


https://websps1.battelle.org/nrcnureg/home/QA

https://websps1.battelle.org/nrcnureg/home

https://websps1.battelle.org/nrcnureg/home/QA

https://websps1.battelle.org/nrcnureg/home/QA/Shared Documents/Forms/AllItems.aspx?View=6156CBC3-5BAE-44FE-B049-37413A4D9AD9

11

Version 2.0 CG Work Plan Activities

Work Plan Activities 2.1 Evaluation and Implementation of Framework Software Improvements 2.2 Evaluation of Advanced Computational Methods 2.3 Evaluation of Classification of Uncertainty 2.4 Improvements to Post Processing/Sensitivity Analyses 2.5 Version 2.0 Framework Software and Module Development

Version 2.0 Coding (Modules & DLLs) 2.6 Create Version 2.0 Beta Code

Version 2.0 Testing (V&V) Complete draft of QA documentation

2.7 Create Final Version 2.0 Code Final QA documentation Final V&V testing completed

Schedule Activities include interim milestones over 27 months to complete by Version

2.0 code December 2013. xLPR Version 2.0 Kickoff Meeting – September 29, 2011– Legacy Hotel, Rockville, MD

SAND#2011-6766P

12

Version 2.0 CG Work Scope - Activity 2.1

Microsoft Access dB with Share Point interface for input parameters that automatically links to framework

Begin draft SQA documentation for framework development

Compile modules as 64-bit DLLs to test for computational improvements

Work with developer to devise requirements for potential framework software modifications

• Improve data exporting by using binary or other efficient file format • Access stored model data in completed run for data analysis without

re-running the code • Enable use of multiple cores using free software player • Incorporate advanced important sampling, adaptive sampling, and/or

post-processing algorithms or add hooks to facilitate use for xLPR xLPR Version 2.0 Kickoff Meeting – September 29, 2011– Legacy Hotel, Rockville, MD

SAND#2011-6766P

13

Version 2.0 CG Work Scope - Activity 2.1 (2)

Test new framework model software features for use in Version 2.0 • Scripting • Optimization and Sensitivity analysis tools • Scenarios • Use new function to select specific percentile values from uncertain

distributions (possible to way to eliminate double looping and switch between constant and uncertain values)


14

Version 2.0 CG Work Scope – Activity 2.2

Study of Advanced Methodologies Available to improve sampling efficiency and solution accuracy.

• Evaluation advanced stochastic sampling approaches Importance sampling Adaptive sampling Stratified sampling Selective lifetime sampling FORM/SORM Methods

• Evaluation optimization algorithms • Evaluation alternative approaches to address the separation of aleatory

and epistemic uncertainties • Must include testing and implementation strategy for xLPR code • Nine month study with letter report documenting results of evaluation,

ranking methodologies for applicability to xLPR for potential inclusion to version 2.0 code.

• May recommend multiple methods for a “tool-kit” approach


15


Evaluation of the Classification of Uncertainty • Support for PIB lead activity to develop a xLPR Guidelines document

for the classification of uncertain parameters for xLPR Provide input and recommendations as needed Review document

• Subtask which includes the evaluation of the framework implementation to ensure consistency with recommendations for xLPR and incorporation of advanced methodologies


16


Improvements to Post Processing/Parameter Sensitivity Analyses • Evaluation of non-linear and non-monotonic responses

• Improve post processing incorporate and test advanced post processing methodologies Integrate post processing into framework

• Improvements to post processing tools and user interface Framework Modules

• Occurs in parallel with Version 2.0 Beta development


17


Create Modules Selected for Version 2.0 • Post Processing Modules • Advanced sampling and optimization Modules

Finalize and code framework software modifications

• Get new version of GoldSim Framework Software

Modify framework to accommodate version 2.0 changes • New modules • Advanced sampling and optimization • Post processing


18

Version 2.0 CG Work Scope - Activity 2.6 & 2.7

Activity 2.6 – Create Version 2.0 Beta • Modify framework to accommodate version 2.0 changes (cont’d) • Revise SQA lifecycle documentation • Develop test cases • Begin V&V framework of functionality • V&V coupled modules • Complete Version 2.0 Beta code • Analyze Version 2.0 Beta code results and debugging

Activity 2.7 – xLPR Version 2.0

• Finalize SQA documentation • Final V&V and integrated test cases • Version 2.0 Release


19

Potential Obstacles, Challenges, & Other Risks

Aggressive Schedule • Nine Month Effort for Advanced Computational Study • Beta Code by end of 2012 • Final Code by end of 2013, fully V&V’d and all SQA documentation

Dependencies on models and input groups • Module features and associated functionality/framework design • Close integration necessary with these groups

Acceptance Group and PIB (define final target) • Development of test cases • Acceptance Criteria for xLPR

Regulatory Requirements Coordination of Staffing Funding of NRC and EPRI contractors

• CR funding and other interruptions will decrease productivity and impact our ability to meet schedule


20

Implementation Schedule Overview

27 Months to complete version 2.0 of the xLPR code

Nine months to study advanced computational methodologies, completed report in July 1, 2012

Must work several activities in parallel • Development of beta version and post processing tools • V&V activities and debugging • SQA documentation will undergo several iterations

Initial prior to beta, interim version at completion of Beta code, and final at version 2.0 release.

Each activity has a lead(s) identified and an expected schedule for completion for each subtask

Delays in an activity can have down stream delays which need to addressed early xLPR Version 2.0 Kickoff Meeting – September 29, 2011– Legacy Hotel, Rockville, MD

SAND#2011-6766P

21

Implementation Schedule – Activity 2.1


22

Implementation Schedule – Activity 2.2


23

Summary

Work Planning and Initial QA Documents xLPR Software Project Management Plan (SPMP) xLPR Software Quality Assurance Plan (SQAP) xLPR Computational Group Work Plan – xLPR-WP-CG1 xLPR Software Configuration Management Plan (SCMP) – under development

Initial Work Plan resulted from recommendations from evaluation of version 1.0 pilot study model and requirements from the xLPR SPMP

Work plan activities will be updated a needed based on results of initial activities and response to upstream changes to modules and down stream requirements

Aggressive Schedule requires integrated team and close interaction with inputs and models groups


xLPR Code Review

Craig Harrington EPRI xLPR Version 2.0 Kick-off Meeting September 29, 2011


2012 International BWR/PWR Materials Conference

International Boiling Water Reactor and Pressurized Water Reactor Materials Reliability Conference &

Exhibition 2012 July 16-19, 2012

Gaylord Resort and Convention Center, 201 Waterfront Street, National Harbor, MD 20745

Email abstract to [email protected] by October 28



xLPR Code Review


ACRS Review Lead - Mike Benson

• Fulfills NRC need to involve ACRS in this sort of project development with regulatory significance

• Excellent source of feedback and insight

• Base proposed frequency is annual

– Met with ACRS Materials Subcommittee – Sept. 21, 2011

– Shorter report to full ACRS likely in November

– Very interested, expressed preference for regular sessions on xLPR elements rather than entire project

• Future details and schedule of ACRS interaction TBD


External Review Lead – Mark Kirk

• Planned in Pilot Study - practical details never worked out • Proposed as:

– Annual interaction – Small, technically diverse expert panel – Should be structured to compliment ACRS Review – May seek international participation

• Challenges: – Selection from many excellent candidates – Funding / contracting arrangements – Defining complimentary review role – Handling administrative role (minutes, reports, etc)


PIB

• Lead – Denny Weakland • Co-lead – Rob Tregoning


Together…Shaping the Future of Electricity

xLPR Acceptance Group

Bob Hardies, NRC Robin Dyle, EPRI

2


Acceptance Group Responsibilities

• Acceptability of the results from the xLPR on a application-specific basis.

• Development of the criteria against which the results from the xLPR code will be compared.

• Application specific technical basis • Application of the results in the form of regulation and/or ASME

code implementation.

3


Specific Applications

• Leak Before Break • Inservice Inspection • GSI 191 • Transition Break Size

4


Risk Metrics

• Rupture • Leakage • Crack growth • Crack initiation

• Output must be consistent with metric

5


Bill

6


Risk Metric Bounds

• For applications based on absolute risk: – Affect of redundant systems – Core Damage Frequency – Large Early Release Frequency – Defense in depth needs to be addressed

• For application based on delta risk – Regulatory Guide 1.174

• Other framework – Risk neutral applications

7


Acceptance Metric Implementation

• Metrics may be implemented differently for different applications – LBB may use RG 1.174 approach for

modifications and operational change evaluation, implemented by relief request

– LBB may use a deterministic approach with risk-informed margins for new applications implemented via Regulatory Guide

– RI-ISI may be implemented via ASME Code – Etc.

8


Issues

• NRC establishes the risk metric • Considerations other than risk

– ALARA – Resources – Personnel safety – Outage sequencing considerations

9


Plan

• Assemble membership – Identify specific applications

• First application = LBB • Establish risk metrics • Identify xLPR output requirements • Implement application • Next application

10


Timeline

June, 2012 target for a draft technical basis for NRR consideration

11


Implementing the xLPR Software Quality Assurance

Program Presented by: Nancy M. Kyle Principal Consultant Theseus Professional Services, LLC “Results Through Service Excellence®” 706-830-3194 / [email protected]










Slide 3

QA Documents


Documents Templates Document Checklists Using Templates and Checklists Document Naming Conventions SharePoint Approval Routing

Slide 4

QA Documents


All xLPR Version 2.0 Project files are stored on the Battelle SharePoint Server https://websps1.battelle.org

Your login takes you directly to the NRCNUREG Site

https://websps1.battelle.org/

Slide 5

QA Documents


All xLPR Version 2.0 QA document templates, checklists, and deliverables are stored on the QA Site

Slide 6

QA Documents


xLPR_QA>QA Documents (Shared Documents)

Slide 7

Document Templates


Templates are being developed for most major software life cycle deliverable documents Individual Work Plan Software Requirements Description Requirements Traceability Matrix Software Verification and Validation Plan Software Design Document Software Test Plan Software Verification and Validation Report Software User Manual

Slide 8

Document Templates


Based on IEEE Software Engineering Standards Provides level of consistency throughout the project Content may be increased, but must include template

elements, as a minimum, for compliance General Structure can evolve under the control of Group

Leads Templates provide a starting point for further refinement to meet

specific needs Understood that templates cannot be a “one-size-fits-all”

Slide 9

Document Templates


xLPR_QA>QA Documents (Shared Documents)>QA Document Templates

Slide 10

Document Checklists


To be completed and included as part of the approval of each document. Provides final check that all required elements are included in the

approved document Includes sign-offs from the Group Lead, Technical Reviewer,

Code Development Lead and SCM Coordinator

Maintained under configuration control along with the document deliverable

Slide 11

Document Checklists


xLPR_QA>QA Documents (Shared Documents)>QA Forms and Checklists

Slide 12

Using Templates and Checklists


Retrieve by double-clicking on the document icon This will open up the document as Read-Only Documents are not to be checked out

Use the MS Word feature of “Save As” to save a copy to your computer.

Create your document using the format of the template If you find that the template doesn’t appear to fit your needs

discuss with your group lead or co-lead Lead and Co-leads can contact QA Administration Group if the

questions can’t be resolved internally

Slide 13

Document Naming Conventions


Documentation shall be named according to the following the pattern: <project>-<acronym of the document>.<extension>

where: <project> = XLPR <acronym of the document> = an acronym that represents the document type and name <doc type-name> <extension>= the appropriate extension of the document type (i.e., .doc, .xls)

Examples XLPR-SCMP.docx for the Software Configuration Management Plan

Document XLPR-WP-CG1.docx for Computational Group Work Plan

A document index is maintained by QA Administration Proposed numbers need to be verified by QA and added to the index

prior to use

Slide 14

Deliverable Storage


Completed project deliverables for Version 2.0 will be stored in the xLPR QA documents site. xLPR_QA>QA Documents (Shared Documents)

Folders are set up for each work group Set up at the work group level Group Lead can set up sub-folder structure

Allows control of documents Displays current version Identifies who last updated and when Identifies if checked out and to whom

Slide 15

Deliverable Storage


Slide 16

Deliverable Storage


Slide 17

Deliverable Storage


Slide 18

SharePoint Approval Routing


Electronic approval process is available in SharePoint Sequential Process with email notification when actions are

required

Deliverables requiring the approval process are uploaded to their storage location before routing for approval.

Approvers are selected from the SharePoint address book

Time constraints can be put on reviews Automatic notification when an approval is overdue

Slide 19



Slide 20



Slide 21



Slide 22



Slide 23



Slide 24



Slide 25



Slide 26

Document Checkout


Checking out a document will produce a local copy of that document on your hard drive Stored in Documents>SharePoint Drafts Creates mapping on your Computer

<folder> on websps1.battelle.org

Checked-out documents are not available for edit by others

Checking in a document increments the version You can choose to discard the checked out document if you don’t

make any changes

Slide 27

Document Checkout


Slide 28

Path Forward


Template development is underway Software Requirements Description is in review Others to follow

SharePoint workflow is being tested with program document approvals Finding that different user privileges provide different capabilities Various firewalls allow different automatic launch capabilities from emails Working to understand these issues and create user guide

Migration from SharePoint to Subversion is being explored

You can always contact QA with questions We are also users of the system but have access to the administrators.

Nothing will hold up work activities


Questions?





Hilda Klasky Oak Ridge National

Laboratory

xLPR Version 2.0 Launch Meeting

September 28-29, 2011

Legacy Hotel Rockville, MD

Subversion Installation and Demonstration


Acknowledgments

• I would like to acknowledge contributions to this presentation from:

• Richard Bass, Nancy Kyle, Dave Rudland, Paul Williams, Patrick Mattie, Brian Zachary.


What is Subversion?

• Subversion is a free/open source version control system (VCS).

• Subversion manages files and directories, and the changes made to them, over time.

• Subversion allows you to recover older versions of your data or examine the history of how your data changed.


Is Subversion the Right Tool?

• If you need to archive old versions of files and directories, possibly resurrect them, or examine logs of how they've changed over time, then Subversion is exactly the right tool for you.

• If you need to collaborate with people on documents (usually over a network) and keep track of who made which changes, then Subversion is also appropriate


Make sure you're not using Subversion to solve a problem that other tools solve

better. • For example, because Subversion replicates

data to all the collaborators involved, a common misuse is to treat it as a generic distribution system.

• People will sometimes use Subversion to distribute huge collections of photos, digital music, or software packages. The problem is that this sort of data usually isn't changing at all.

• The collection itself grows over time, but the individual files within the collection aren't being changed. In this case, using Subversion is “overkill.”


To begin, create ORNL XCAMS account In order to have access to the evaluation version of Subversion xLPR repository, users need to create an XCAMS account at: https://xcams.ornl.gov/xcams/


To be granted access to the xLPR Subversion Repository, you will

need permission from ORNL.

After creating your XCAMS account, please send an email, with your new user id, to Hilda Klasky at [email protected].



Subversion Installation

• The Subversion Repository has already been installed and configured at a server located in the Open Research Zone at ORNL

• Proper tuning will follow in the subsequent weeks.

• Users need to install Subversion clients (for example, TortoiseSVN) to access the server.


Subversion Repository Directory Structure has Three Top-Level

Directories • The project's tree structure contains three top-level

directories named branches, tags, and trunk. The trunk directory should contain all of your data, and initially ,the branches and tags directories should be empty :

/svn/ test/

branches/ tags/ trunk/

foo.c Makefile …


Install an SVN Client: TortoiseSVN

Download svn client

(for example, I am using: TortoiseSVN from http://tortoisesvn.net/downloads.html)

NOTE: If you are blocked from downloading due to security issues, then right-click on “OPTIONS” and select “download file”

http://tortoisesvn.net/downloads.html

Initiate download of SVN Client by clicking on “TortoiseSVN 64-Bit”

(or 32-Bit)

Next, verify that you have a good download file by checking the digital

signature!


The next step is to install TortoiseSVN on your computer

Installation of TortoiseSVN client requires that you • have admin rights and • re-start your computer after installation


With the TortoiseSVN client installed on your computer, you can now explore the features of the Subversion Repository

In the following slides, I will describe application of the SVN client to the Subversion Repository

The XLPR test subversion repository can be “checked out”, i.e., a copy can

be downloaded to your computer 1. Select a target folder for copy of repository


be downloaded to your computer 2. Right-click on target folder to access “SVN Checkout”


be downloaded to your computer 3. Click on “SVN Checkout” to get the “Checkout” box, then click on “OK”


be downloaded to your computer 4. Clicking on “OK” in the “Checkout” box leads to the “Authentication” screen


be downloaded to your computer 5. Enter your XCAMS user id and password into the “Authentication” screen and click on “OK”

Following authentication, a copy of the repository is ported to target file

on your computer • uthen


TortoiseSVN works through Icon overlays and context menus

• Icon Overlays:


Examples of frequently encountered icons are those described below

– A fresh checked out working copy has a green checkmark as overlay. That means the Subversion status is normal.

– As soon as you start editing a file, the status changes to modified and the icon overlay then changes to a red exclamation mark. That way you can easily see which files were changed since you last updated your working copy and need to be committed.

– If during an update, a conflict occurs, then the icon changes to a yellow exclamation mark.


Examples of frequently encountered icons are those described below (continued)

– This icon shows you that some files or folders inside the current folder have been scheduled to be deleted from version control or a file under version control is missing in a folder.

– The plus sign tells you that a file or folder has been scheduled to be added to version control.

– The bar sign tells you that a file or folder is ignored for version control purposes. This overlay is optional.

– This icon shows files and folders which are not under version control, but have not been ignored. This overlay is optional


TortoiseSVN (cont.)

• Context Menus:

The “normal” icon shown on file “test” indicates local image of repository is

identical to that on server • ormnal

As soon as you start editing a file, status changes to modified and icon overlay then

changes to red exclamation mark

Right click on the modified file to reach the “SVN Commit” command

Click on “SVN commit” command to reach commit screen; enter message

describing change

Clicking “OK” on Commit screen completes the action and “modified” icon

changes to “normal” on variables_list


Now that we have seen the “commit” process, lets review a typical work cycle

The typical work cycle looks like this:

1. Update your working copy.

2. Make your changes.

3. Review your changes.

4. Fix your mistakes.

5. Resolve any conflicts (merge others' changes).

6. Publish (commit) your changes.

To update the working copy, right-click on “test” file and select “SVN Update”


Updating your working copy will bring changes from other clients committed

to repository into your copy


The SVN client allows users to compare their working copy with that copy found

in the repository

• Viewing Differences: – Local changes If you want to see what changes you

have made in your working copy, just use the explorer context menu and select TortoiseSVN -> Diff.


Multiple users working on the same file can lead to conflicts that must be

resolved

• Resolving Conflicts

• There are two kinds of conflicts: – File conflicts: A file conflict occurs if two (or

more) developers have changed the same few lines of a file.

– Tree conflicts: A tree conflict occurs when a developer moved/renamed/deleted a file or folder, which another developer either also has moved/renamed/deleted or just modified.


A sample file conflict is presented with the conflicting area marked as shown

below File Conflicts The conflicting area is marked like this: <<<<<<< filename

your changes

=======

code merged from repository

>>>>>>> revision

Subversion places three additional files in your directory: – filename.ext.mine

– filename.ext.rOLDREV

– filename.ext.rNEWREV


To further extend your understanding of these many important features, here is some

suggested reading on Subversion and TortoiseSVN

1. Read the Subversion free book at: http://svnbook.red-bean.com/

2. Read the Tortoisesvn free documentation at: http://cdnetworks-us-2.dl.sourceforge.net/project/tortoisesvn/1.6.16/Documentation/TortoiseSVN-1.6.16-en.pdf

http://svnbook.red-bean.com/

http://cdnetworks-us-2.dl.sourceforge.net/project/tortoisesvn/1.6.16/Documentation/TortoiseSVN-1.6.16-en.pdf





Questions?