Lesson12: Supportability Design Reviews - DAU · Web view2012/12/12 · LOG 211 Supportability Analysis Student Guide LOG 211 Supportability Analysis Student Guide January 2013 Final

LOG 211 Supportability Analysis

Student Guide

LOG 211 Supportability Analysis

Student Guide

Lesson12: Supportability Design Reviews

Content

Slide121. Lesson 12: Supportability Design Reviews

Welcome to Lesson 12 on Supportability Design Reviews.

Introduction

Content

Slide 122. Topic 1: Introduction

Comment by PDallosta: Slide 12-3 page 12-3 update TMRR

Technology Maturation & Risk Reduction

Slide 123. Life Cycle Management Framework: Where Are You? What Influence Do You Have?

Supportability design reviews offer Life Cycle Logistician’s (LCL’s) opportunities to impact a system, from “Influencing the Design” to “Supporting the Design” in the life cycle. There are numerous design reviews throughout the Life Cycle Management Framework, but for the purposes of this lesson, the Supportability design reviews listed below will be covered.

Materiel Solution Analysis Phase

Alternative Systems Review (ASR)

Technology Development Maturation and Risk Reduction (TMRR) PhaseComment by PDallosta: Updated TMRR

System Functional Review (SFR)

Preliminary Design Review (PDR)

Engineering & Manufacturing Development Phase

Critical Design Review (CDR)

Developmental Test and Evaluation (DT&E)

Functional Configuration Audit (FCA)

Production Readiness Review (PRR)

Production & Deployment Phase

Initial Operational Test and Evaluation (IOT&E)

Physical Configuration Audit (PCA)

Content

Slide 124. Topics and Objectives

Content

Slide 125. Supportability Design Reviews Lesson Approach

Key questions to ask for each review in this lesson are:

What inputs are necessary for a Supportability design review at this phase in the Life Cycle Management Framework?

How does this particular design review impact the system’s Supportability?

What must be achieved before this review is considered complete?

Overview of Supportability Design Reviews

Content

Slide 126. Topic 2: Overview of Supportability Design Reviews

Comment by PDallosta: Slide 12-7 page 12-7 update to TMRR


Slide 127. What Are Supportability Design Reviews?

Before delving into each specific Supportability design review, it is helpful to understand how reviews are used at the program level. From the perspective of the Program Manager, these reviews are an opportunity to ascertain whether or not certain criteria (described as “exit criteria” in this lesson) have been met and whether a decision to proceed is warranted. In general, the LCL supports these reviews through the Supportability analyses discussed throughout this course. The LCL should participate in a review to explain how those analyses support a decision as to whether the exit criteria have been met and the program can proceed.

In summary, these reviews may:

Serve to confirm major technical efforts within the Life Cycle Management Framework

Verify the outputs from each phase within the Life Cycle Management Framework

Ascertain progress in defining system technical requirements

Ensure that the system under review has a reasonable expectation of being judged operationally effective and suitable

Be required by contract and/or DoD policy to include review processes and requirements

Content


Slide 128. Supportability Design Reviews: Overview

In general, Supportability design reviews follow the same basic pattern:

1. An event serves as an impetus for a review conference to convene. Note: Although most reviews are part of a master schedule, the reviews are conducted to determine if the criteria have been met, not when the criteria have been met. This can be refered to as “event-driven” as opposed to “schedule-driven.”

The review consists of the IPTs and other stakeholders applying various criteria and Supportability questions.

Upon successfully completing all actions associated with a particular review, the system moves to the next review, development phase and/or project milestone. Often the outputs are used in the next review.

Although each design review requires specific entry/exit criteria to make particular decisions, in general, they include the inputs, processes and outputs described below.

Supportability Design Review Requirements

Entry Criteria: Phase-specific accomplishments required to be completed before a program is allowed to enter a particular phase in the Life Cycle Management Framework

The program may have to fulfill criteria, including:

Maturity

Performance

Documentation

Inputs: Standup Integrated Product Teams (IPTs)

Identify role: Who is doing what?

Define design review goal

Define schedule/timeline

Supportability Considerations/Questions:

Affordability?

Effective Design?

Supportability?

Functionality?

Design Readiness?

Design Capability?

Producibility?

Outputs: Integrated Product Teams

Assess the system’s Supportability requirements and characteristics for compliance to the review’s entrance and exit criteria

Identify Supportability deficiencies, identify potential impacts to the design and make recommendations regarding their resolution

Identify requirements and mechanism for exchanging information, to include analyses, reports, and data

Assign action items, specifying responsibilities, schedule and expected outcome(s)

Exit Criteria: Program-specific accomplishments that must be satisfactorily demonstrated before a program can progress further in the current Life Cycle Management Framework or transition to the next phase.

Specific Supportability Design Review Requirements, the Defense Acquisition Guidebook and Assessment Checklists

Inputs/outputs, entry/exit criteria, and Supportability questions/considerations specific to a particular review are discussed in the next topic. In addition Chapters 4, 5, and 9 of the Defense Acquisition Guidebook (DAU) provide detailed discussion of technical reviews requirements for Systems Engineering, Life Cycle Logistics and the Test & Evaluation communities.

Content

For example, Section 4.3, Systems Engineering Activities In the Life Cycle, outlines the Technical Review process. To assist in the preparation for and conduct of technical reviews technical review risk assessment checklists are available for each of the reviews. These checklists are designed as a technical review preparation tool, and should be used as the primary guide for the risk assessment during the review. The checklist itself can be both an input to, and an output of, the review. The Integrated Product Team can do a self-assessment as input, but the review participants should take ownership of the assessed risk as an output of the review. These checklists are available on the Systems Engineering Community of Practice and are accessible in the DAU Technical Reviews Continuous Learning Module, CLE-003.

Content

Slide 129. Design Reviews and Affordability

The logistician’s Supportability and life cycle goals during Supportability design reviews should:

Leverage and use performance-based logistics strategies to reinforce optimization of total system availability while minimizing cost and logistics footprint

Merge design effectiveness and process efficiency to gain the maximum mission effectiveness as the goal for performance-based logistics

Offer opportunities to check how the system is developing along effective lines of Affordability and Supportability

Supportability Design Reviews and Criteria

Content

Slide 1210. Topic 3: Supportability Design Reviews and Criteria


Slide 1211. Alternative Systems Review (ASR)

Definition

In the Materiel Solutions Analysis phase, the Alternative Systems Review (ASR) is a technical review that demonstrates the preferred concept is cost effective, affordable, operationally effective, and suitable, and can be developed to provide a timely solution to a need at an acceptable level of risk. This review also determines whether the system is ready to proceed to the next phase, Technical Development.

By reviewing alternative materiel solutions, the ASR helps ensure that sufficient effort has been given to conducting trade studies that consider and incorporate alternative system designs that may more effectively and efficiently meet the defined capabilities. A successful review is predicated on the Integrated Product Team's determination that the operational capabilities, proposed solution(s), available technologies, and program resources (funding, schedule, staffing, infrastructure, and processes) form a satisfactory basis for proceeding into the Technology Development Maturation and Risk Reduction (TMRR) phase.Comment by PDallosta: Updated TMRR

Key ASR Question

The Alternative Systems Review answers many questions that will be addressed next. The key ASR question asks whether one or more of the proposed systems are cost-effective, affordable, operationally effective and suitable with an acceptable level of risk. The LCL can address this key question using the inputs, entry criteria, Supportability questions, exit criteria and outputs discussed with the following slide.

Content

Slide 1212. ASR Process

Inputs

The Alternative Systems Review begins with the following inputs:

Successful completion of:

Initial Technical Review (ITR)

Program System Review (PSR)

A data repository that includes:

Preferred System Solution(s) concept/description

Draft Request for Proposal (RFP)

Analyses results and definitions

System Requirements document

Including interoperability and system distributed services requirements

Initial Capabilities Document (ICD)

Alternative maintenance and Sustainment concepts

Updated cost and schedule data

In addition to these inputs, the following key entry questions and/or issues need to be addressed:

Is there a plan to conduct an ASR, or has one been completed in preparation for a Milestone A Decision?

In preparation for the ASR, what alternative systems were evaluated during the Capabilities Assessment phase?

Do the stakeholders have an understanding of how available system concepts meet capabilities described in the ICD, and of the Affordability, operational effectiveness and technology risks inherent in each alternative concept?

Supportability Considerations/Questions

During the ASR, you should consider questions or criteria under the following key questions:

1. Affordability? Including support costs, is the system cost effective?

Effectively Designed? Is it operationally effective and suitable?

Supportability? Can it be supported in the field to do its mission?

You can find more specific Supportability questions and considerations for the Alternative Systems Review in the Lesson 12 Job Aid that is located after the lesson Summary.

Outputs

ASR is successful when it accomplishes and/or produces the following exit criteria and outputs:

Preliminary system specifications

Systems Engineering Plan (SEP)

Support and maintenance concepts

The following questions must also be successfully answered:

1. Is the system software scope and complexity sufficiently understood and addressed in the Technology DevelopmentAcquisition Strategy to enable low software technical risk?

Has the system technical baseline been captured in a preliminary system specification that is consistent with technology maturity and the proposed program cost and schedule?

(Source: DAG, p. 230)

Successful completion of the Alternative Systems Review leads to the passing of MS A and the start of the Technology Development Maturation and Risk Reduction (MRR) phase.Comment by PDallosta: Updated TMRR

Content


Slide 1213. System Functional Review (SFR)

Definition

In the Technology Development Maturation and Risk Reduction (TMRR) phase, the System Functional Review (SFR) is a formal review of the conceptual design of the system to establish its capability to satisfy requirements. It establishes the functional baseline and seeks to confirm that the system has a reasonable expectation of satisfying requirements.

Key SFR Question

The System Functional Review answers many questions that will be addressed next. The key SFR question asks whether the system can reasonably satisfy the requirements of the Initial Capabilities Document.

The LCL can address this key question using the inputs, entry criteria, Supportability questions, exit criteria and outputs discussed with the following slide.

Content

Slide 1214. SFR Process

Inputs

The System Functional Review begins with the following entry criteria and inputs:

Draft Capability Development Document (CDD)

Requirements traced from CDD to system specifications, functional specifications and subsystem specifications

Approved Materiel Solution

Preliminary Technical Performance Measurements (TPMs) identified and traceable to Measures of Effectiveness (MOE) and Measures of Performance (MOP)

Functionality assigned to Key Performance Parameters (KPPs)

Verification and validation methodology defined for each specification requirement

System functional specification completed to subsystem level

Functional requirements for operations and maintenance assigned to subsystems, hardware, software, or support after detailed reviews of the design’s architecture

Updated

Systems Engineering Plan

Software Development Plan

Logistics and training requirements identified in system and subsystem specifications

Additional inputs:

Updated Integrated Architecture Viewpoints

Updated use case analysis

Completed Preliminary subsystem specification and Interface Design Documents

Allocated Information Assurance Controls over to system components

Created Technology maturation plans for critical technologies.

Defined Technology Maturation and Risk Reduction (TMRR) Development Exit Criteria.


During the SFR, you should consider questions or criteria under the following key questions:

1. Affordability: Is the program with the approved functional baseline executable within the existing budget?

Functionality: Are the system functional requirements sufficiently detailed and understood to enable system design to proceed?

Supportability: Are the Supportability requirements to achieve the support strategy included in the performance specifications?

You can find more specific Supportability questions and considerations for the System Functional Review in the Lesson 12 Job Aid.

Outputs

SFR is successful when it accomplishes and/or produces the following exit criteria and outputs:

Establishes:

System functional baseline with traceability to lower-level performance requirements

Preliminary system-level maintenance plan

Updates:

Cost Analysis Requirements Description (CARD), or equivalent

Test and Evaluation Management Plan (TEMP), or equivalent

Program development schedule including system and software critical path drivers

Risk assessment for the EMD phase

(DAG, pp. 243-5)

Successful completion of the System Functional Review leads to the Preliminary Design Review.

Content


Slide 1215. Preliminary Design Review (PDR)

The Preliminary Design Review (PDR) is the next Pre-Milestone B review we cover in the Technology Maturation and Risk Reduction (TMRR) Development phase. The PDR influences system design and support by ensuring that the allocated baseline is established. Each function in the functional baseline is allocated to one or more system configuration items to ensure traceability.

Definition

The PDR is a formal review that confirms the preliminary design logically follows the SFR findings and meets the requirements. It normally results in approval to begin detailed design.

Key PDR Question

The Preliminary Design Review answers many questions that will be addressed next. The key PDR question asks whether the system has a reasonable chance of being judged operationally effective, suitable and affordable before becoming a Program of Record (POR) at Milestone B.


Content

Slide 1216. PDR Process

Inputs

The Preliminary Design Review begins with the following entry criteria and inputs:

Approved Life Cycle Sustainment Plan with updated program Sustainment development efforts and schedules

Subsystem specifications for hardware, software, and associated Interface Control Documents. These enable detailed design or procurement of subsystems.

System requirements/capabilities

Test and Evaluation Master Plan (TEMP), including:

Sell-off criterion for each product

Witness requirements

Documentation

Cost Analysis Requirements Description (CARD), updated as required

Risk and opportunity management, reviewed and managed regularly

Configuration Management Board running with metrics

In addition to these inputs, the following key entry issues need to be addressed. In preparation for the PDR, are the following activities/actions completed or documents/information available?

Requirements

Design Reliability

Design Maintainability

Requirements Traceability Matrix

Data

Supplier data describing specific components

Design data defining major subsystems, equipment, software and other system elements

Design & Analyses

Reliability & Maintainability Analyses, reports, trade studies

Supportability analysis data/Logistics Product Database

Design documentation

Equipment and part standardization

Costs/Risks

Developmental Test & Evaluation plans

Spares and Government-Furnished Property (GFP)


During the PDR, you should consider questions or criteria under the following key questions:

1. Affordability: Is the program with the approved functional baseline executable within the existing budget?

Design Readiness: Have the majority of manufacturing processes been defined and characterized?

Supportability: Does the system meet all logistics and Supportability requirements?

You can find more specific Supportability questions and considerations for the Preliminary Design Review in the Lesson 12 Job Aid.

Content

Outputs

PDR is successful when it accomplishes and/or produces the following exit criteria and outputs:

Establishes:

Preliminary Design Review (PDR) Report

A system allocated baseline

The Requirements Traceability Matrix (RTM)

Updates:

Cost Analysis Requirements Description (CARD) or CARD-like document based on the system allocated baseline

Risk assessment for Engineering and Manufacturing Development (EMD)

Approved Life Cycle Sustainment Plan updating program

Sustainment development efforts and schedules

Program schedule, including system and software critical path drivers, and hardware, software, human/support systems

(DAG, pp. 247-50)

Successful completion of the Preliminary Design Review is a prerequisite for a successful Milestone B decision, which permits the system to enter the Engineering and Manufacturing Development phase.


Slide 1217. Critical Design Review (CDR)

Definition

In the Engineering and Manufacturing Development phase, the Critical Design Review (CDR) is a formal review conducted to evaluate the completeness of the design and its interfaces. In addition to establishing the initial product baseline and placing it under configuration control, the CDR also ensures the system under review has a reasonable expectation of satisfying the requirements of the Capability Development Document (CDD) within the current budget and schedule.

Key CDR Question

From a Supportability perspective, the Critical Design Review answers the question of whether the system meets all logistics and Supportability requirements from the Capability Development Document within the current allocated budget and schedule.


Content

Slide 1218. CDR Process

Inputs

The Critical Design Review begins with the following entry criteria and inputs:

Approved Life Cycle Sustainment Plan that updates program Sustainment development efforts and schedules

System requirements and capabilities

CARD, updated as required

Risk and opportunity management reviewed and managed regularly

Product baseline under configuration control

In addition to these inputs, the following key entry issues need to be addressed. In preparation for the CDR, are the following activities/actions completed or documents/information available?

Design

Prepare production baseline (“Build To” documentation)

Determine if the system design documentation (product baseline, including Item Details Specs, Material Specs, Process Specs) is satisfactory to start initial manufacturing

Capture product baseline in the detailed design documentation

Review test plans to assess if test efforts are developing sufficiently to indicate the Test Readiness Review will be successful

Costs/Risk

Produce published agenda (several days prior to the conference)

Prior Reviews

Completed all action items related to the previous conference (PDR) successfully


During the CDR, you should verify the achievement of the following criteria:

1. Affordability: Is the program executable with the existing budget and the approved product baseline?

Design Capability: Does the detailed design satisfy the Capability Development Document or any available draft Capability Production Document?

Supportability: Does the system meet all logistics and Supportability requirements?

You can find more specific Supportability questions and considerations for the Critical Design Review in the Lesson 12 Job Aid.

Outputs

CDR is successful when it accomplishes and/or produces the following exit criteria and outputs:

Establishes:

Post-CDR Report

System initial product baseline is established and places it under configuration control

Issues and Actions Summary

Completed FMECA/FTA/ Safety and Maintainability analyses

Estimates of System R&M

Supportability requirement enablers including Human Systems Integration (HSI) design features, with Environment, Safety and Occupational Health (ESOH) risk reduction features included in the design

Updates:

Cost Analysis Requirements Description (CARD) or CARD-like document based on the system product baseline.

Risk Assessment for the EMD phase, including issues/risks that could result in a breach to the program baseline or substantively impact cost, schedule or performance.

Approved Life Cycle Sustainment Plan, including:

Updated program Sustainment development efforts

Schedules based on current budgets

Test evaluation results

Supportability design features Developmental Testing Results confirmed for key Sustainment enablers or drivers requirements are complete. If the test results to date do not indicate the operational test is likely to succeed or risk has increased, new developmental and operational testing criteria and plans should be considered, along with fallback plans.

(DAG, pp. 262-5)

Successful completion of the Critical Design Review leads to the next review in the Engineering and Manufacturing Development phase.

Content


Slide 1219. Developmental Test and Evaluation (DT&E)

Developmental Test and Evaluation (DOT&E) activities are principally conducted during the Technology Maturation and Risk Reduction (TMRR) Development (TD) and Engineering and Manufacturing Development (EMD) phases to ensure the system design satisfies desired capabilities.

Definition

DT&E is an engineering test process conducted to provide data for the assessment of a design’s achievement of critical system performance parameters. The testing is performed at the component, subsystem and system-level configurations of hardware and software, and ensures the verification and validation of the Systems Engineering process.

DT&E is conducted as part of the design process. The testing may be conducted by the Contractor under Government oversight, independently by the Government, or jointly by the parties. DT&E differs from Initial Operational Test and Evaluation (IOT&E) in that the IOT&E is conducted by the Service’s Operational Test Agency (OTA) and determines the system’s operational effectiveness and suitability in the operational environment.

Key DT&E Question

Developmental Test and Evaluation (DT&E) answers the question of whether the system design solution satisfies desired capabilities. The LCL can address this key question using the inputs, entry criteria, Supportability questions, exit criteria and outputs discussed with the following slide.

Content

Slide 1220. DT&E Process

Inputs

Developmental Test and Evaluation begins with the following entry criteria and inputs:

Areas needing test to complete FMECA/FTA/RCM Analysis

Testing required as part of Reliability improvement program

Performance estimates and specifications

In addition to these inputs, the following key entry questions need to be addressed:

Are Modeling and Simulation (M&S) uses in Developmental Test and Evaluation (DT&E), Operational Test and Evaluation (OT&E), Live Fire Test and Evaluation (LFT&E), System of Systems (SoS) interoperability testing and information assurance testing integrated in an efficient continuum?

Has the system demonstrated sufficient technical maturity into Initial Operational Test and Evaluation (IOT&E) regarding Critical Technical Parameters (CTPs), including interoperability, and is it documented in the Test and Evaluation Master Plan (TEMP)?


During DT&E, you should focus on the following key question:

Supportability: Are Supportability metrics being met?

You can find more specific Supportability questions and considerations for the Developmental Test and Evaluation in the Lesson 12 Job Aid.

Outputs

DT&E is successful when it accomplishes and/or produces the following exit criteria and outputs:

Areas requiring redesign

Improvements in Reliability

Successful completion of the Developmental Test and Evaluation leads to the next review in the Engineering and Manufacturing Development phase, the Functional Configuration Audit (FCA).

(DAG, pp. 802-3)

Content


Slide 1221. Functional Configuration Audit (FCA)

Definition

The Functional Configuration Audit (FCA) is the second review in the EMD phase and may be conducted concurrently with the System Verification Review (SVR). SVR is a multi-disciplined product and process assessment that ensures the system under review can proceed into Low-Rate Initial Production.

The FCA is a formal review conducted to verify that all subsystems can perform all of their required design functions in accordance with their functional and allocated configuration baselines.

Key FCA Question

The Functional Configuration Audit answers the question of whether the functionality or performance of hardware and software configuration items meets their specifications.


Content

Slide 1222. FCA Process

Inputs

The Functional Configuration Audit begins with the following entry criteria and inputs:

Accomplishment of program Supportability objectives detailed in acquisition documents (CDD, LCSP)

A completed system Technical Data Package

Developmental Test & Evaluation (DT&E) results

In preparation for the FCA, are the following activities/actions completed or documents/information available?

Requirements:

Functional and allocated baselines

Verification is comprehensive and complete

Testing:

Test procedures and results

Preproduction and production test results

Design & Analyses:

Configuration audits, including completion of all change actions, have been completed for all configuration items

Systems engineering planning is updated for production

Costs/Risks:

Critical achievements, success criteria and metrics have been established for production

Risk management planning has been updated for production

Readiness issues for continuing design, continuing verifications, production, training, deployment, operations, support and disposal have been resolved


During FCA, you should focus on the following key question:

Functionality: Do all the subsystems do what they are supposed to do according to the functional configuration?

Outputs

FCA is successful when all FCA-related actions and risks are identified, resolved or effectively mitigated to an acceptable level, and the EMD product is sufficiently mature for entrance into Low-Rate Initial Production.

Successful completion of the Functional Configuration Audit leads to the next review in the Engineering and Manufacturing Development phase, the Production Readiness Review (PRR). (DAG, pp. 267-8)

Content


Slide 1223. Production Readiness Review (PRR)

Definition

The Production Readiness Review (PRR) is the last review covered in the EMD phase. The PRR is a formal examination of a program to determine if:

The design is ready for production

Production engineering problems have been resolved

The producer has accomplished adequate planning for the Production and Deployment phase

Key PRR Question

The Production Readiness Review answers the question of whether the system design is ready for production.


Content

Slide 1224. PRR Process

Inputs

The Production Readiness Review (PRR) begins with the following entry criteria and inputs:

Long lead-time items identified and ordered

Mature software

Manufacturing processes identified and validated

Special tooling: jigs, fixtures and other manufacturing aids available

Quality assurance procedures and methods in place

Manufacturing personnel trained

In addition to these inputs, the following key entry issues need to be addressed. In preparation for the PRR, are the following activities/actions completed or documents/information available?

Design

The developer’s design is complete.

Product design is low risk from the standpoint of producibility and has stabilized.

The design is in consonance with the operational, maintenance and support concepts, including meeting inter-service and foreign interoperability requirements.

Costs/Risk

Verification of the design has been accomplished, including qualification of subsystems and components as appropriate, and demonstration of performance and R&M characteristics.

Long lead-time materials identified, and action initiated for advance procurement where appropriate. Sole source items are identified, and continuity of supply is ensured.

Production cost projections have been made and are well supported.

Data

The Technical Data Package will permit competitive acquisition and domestic and foreign co-production, where appropriate.

Prior Reviews

A Critical Design Review has been accomplished and discrepancies resolved.

Training and Manpower

Skilled production manpower will be available in sufficient numbers for the planned term of production.


During the PRR, you should focus on the following key question:

Producibility: Are we ready to go into production? Producibility is the degree to which “Design for Manufacturing” concepts have been used to influence system and product design. Producibility facilitates timely, affordable, and optimum-quality manufacture, assembly, and delivery of system to the field. You can find more specific Supportability questions and considerations for the Production Readiness Review in the Lesson 12 Job Aid.

Outputs

PRR is successful when it accomplishes and/or produces the following exit criteria and outputs:

IPT reviews readiness of manufacturing process, including:

Quality management system

Production planning (i.e., facilities, tooling and test equipment capacity, personnel development and certification, process documentation, inventory management, etc.)

IPT determines:

System requirements are fully met in the final production configuration.

Production capability forms a satisfactory basis for proceeding into Low-Rate Initial Production (LRIP) and Full-Rate Production.

Remaining risks have been captured and risk mitigation plan is in place.

Successful completion of the Production Readiness Review leads to the next review in the Engineering and Manufacturing Development phase. (DAG, pp. 268-9)

Content


Slide 1225. Life Cycle Management Framework: Where Are You?

What Influence Do You Have?

You have reached the Production and Deployment phase of the course. You will learn about two Supportability design reviews that occur during this phase. First, let’s examine the general characteristics of the Production and Deployment phase.

Slide 1226. Production and Deployment Phase

The Production and Deployment phase commences at Milestone C. During the Production and Deployment phase, the system should achieve operational capability that satisfies mission needs.

Inputs to Production and Deployment include:

Evaluation results from testing

Identification of:

Exit criteria for the Production and Deployment Phase

Entry criteria for the Operations and Support Phase

Acquisition program baseline

Capability Development Document and Capability Production Document

Systems Engineering Plan

Test and Evaluation Master Plan

Programmatic Environment, Safety, and Occupational Health Evaluation

Product support element requirements

System Safety Analyses to include updated ESOH Risk Analysis (section 4.4.7.5), update to Safety Requirements Criteria Analysis, System Safety Hazard Analysis

Content

Two work efforts take place during the Production and Deployment phase:

Low-Rate Initial Production (LRIP)

Full-Rate Production and Deployment (FRP&D)

(DAG, p. 271)

Content


Slide 1227. Initial Operational Test and Evaluation (IOT&E)

Definition

Initial Operational Test and Evaluation (IOT&E) is the last review covered in this lesson. Earlier we discussed DT&E, which the design team conducted. IOT&E is an evaluation of operational effectiveness and Suitability made by an independent operational test agency, with user support as required.

Key IOT&E Question

The Initial Operational Test and Evaluation answers the question of whether the system performs in an operationally effective and suitable manner under realistic operational conditions.


Content

Slide 1228. IOT&E Process

Inputs

The Initial Operational Test and Evaluation (IOT&E) begins when the developing organization provides the following inputs to an independent testing authority:

System Reliability, Maintainability and Supportability performance requirements, demonstrated in as realistic an operating environment as possible

Completed:

Life Cycle Sustainment Plan

Systems Engineering Report

Logistics Product Database

Test and Evaluation Master Plan

Technical manuals

Training plan/program

Provisioning plan

Product support strategy

Content


The Initial Operational Test and Evaluation process works differently than the reviews discussed previously. In IOT&E, the developing organization hands off the inputs listed above to an independent agency who reports to Congress. The independent agency will compile a list of deficiencies and trouble tickets. The LCL must address and resubmit the fixed deficiencies to the independent agency.

During the IOT&E, the independent agency will be answering the following questions:

Supportability: Does the system satisfy the user’s needs?

Suitability: Does it work as advertised including Supportability?

You can find more specific Supportability questions and considerations for the Initial Operational Test and Evaluation in the Lesson 12 Job Aid.

Outputs

When the independent agency finds that the deviances and related issues have been solved the system can proceed out of IOT&E with the following outputs:

Issued the Beyond Low Rate Initial Report (BLRIR) to Congress

Measured the system’s operational effectiveness and suitability performance in operational use.

Successful completion of the Initial Operational Test and Evaluation leads to the next review in the Production and Deployment phase. (DAG, pp. 811-3)

Content


Slide 1229. Physical Configuration Audit (PCA)

Definition

The Physical Configuration Audit (PCA) is a formal audit that establishes the final product baseline as reflected in an early production configuration item, and is conducted before the decision for Full-Rate Production.

A configuration item (CI) is any item that satisfies an end-use function and is designated for separate configuration management. Any item required for Product Support and designated for separate procurement is a CI

The Physical Configuration Audit answers the question of what is the actual configuration of an item being produced; it is the final check before hand-off to Sustainment.

Please note: The PCA risk assessment checklist is designed as a technical review preparation tool, and should be used as the primary guide for assessing risk during the review. This checklist is available on the Systems Engineering Community of Practice.

Key PCA Questions

What is the actual physical configuration of the item being produced?

Is the item accurately reflected in the LPD?

The LCL can address these key questions using the inputs, entry criteria, Supportability questions, exit criteria and outputs discussed with the following slide.

Content

Slide 1230. PCA Process

Inputs

PCA begins with the following inputs, entry questions and/or issues:

Complete Technical Data Package, describing the product baseline

Completed manufacturing and quality control plans


During PCA, you should focus on the following key questions:

Producibility: Do we have a clear picture of the total configuration of the system? Do we know all the parts in the system? Can we produce at full rate production?

Outputs

PCA is successful when it accomplishes and/or produces the following exit criteria and outputs:

All PCA-related actions and risks are identified, resolved or effectively mitigated to an acceptable level.

The design and manufacturing documentation match the item as specified in the contract.

Signed PCA certificate of completion or approval between the developing and the sustaining organizations that is submitted to the Milestone Decision Authority (MDA) for Full Rate Production(FRP)

(DAG, pp. 274-5)

Content

Slide 1231. IPS Elements and IPTs

The Integrated Product Teams conduct Supportability Design Reviews to:

Assess the system’s Supportability requirements and characteristics for compliance to the review’s entrance and exit criteria

Identify Supportability deficiencies, identify potential impacts to the design and make recommendations regarding their resolution

Identify requirements and mechanism for exchanging information, to include analyses, reports, and data

Assign action items, specifying responsibilities, schedule and expected outcome(s)

The IPTs most involved in developing the design interface for Supportability include:

Human Systems Integration

Test & Evaluation

Product Support Management

Systems Engineering

These teams will be responsible for providing updates to documents such as the Life Cycle Sustainment Plan, the Systems Engineering Plan, the Test and Evaluation Master Plan, the RAM-C Rationale Report and others.

Exercise

Content

Slide 1232. Topic 4: Exercise

Slide 1233. Exercise

Purpose

The purpose of the exercise is for students to apply their understanding of logistics Supportability issues during design reviews. Students will differentiate impacts that Supportability design reviews have on Supportability and Supportability Analysis.

Instructions

1. Log into Blackboard and navigate to the Lesson 12 Exercise.

Answer the questions.

Submit your answers in Blackboard.

Summary

Content

Slide 1234. Topic 5: Summary

Content

Slide 1235. Takeaways

Content

Slide 1236. Summary

Supportability Design Reviews Job Aid

The purpose of this job aid is to provide the student with an organized collection of specific Supportability questions and considerations for each of the Supportability Design Reviews covered in Lesson 12: Supportability Design Reviews.

Supportability questions and considerations for the following Supportability design reviews are covered:

Materiel Solution Analysis Phase

Alternative Systems Review (ASR)

Technology Maturation and Risk Reduction (TMRR) Development Phase

System Functional Review (SFR)

Preliminary Design Review (PDR)

Engineering & Manufacturing Development Phase

Critical Design Review (CDR)

Developmental Test and Evaluation (DT&E)

Functional Configuration Audit (FCA)

Production Readiness Review (PRR)

Production & Deployment Phase

Initial Operational Test and Evaluation (IOT&E)

Physical Configuration Audit (PCA)

Alternate Systems Review

Affordability: In addition to answering the question about the cost-effectiveness of the system, you should also ask:

Are the risks for competitive prototyping and initial development (through to the allocated baseline) known and manageable?

Have required investments for technology developmentmaturation, to mature design and manufacturing related technologies, been identified and funded?

Effectively Designed: In addition to asking whether the system is operationally effective and suitable, you should also ask:

Is the proposed materiel solution(s) sufficiently detailed and understood to enable entry into Technology Maturation and Risk Reduction (TMRR) Development with low technical risk?

Are the system software scope and complexity sufficiently understood and addressed in the planning for the Technology Maturation and Risk Reduction (TMRR) Development phase to enable an acceptable/manageable level of software technical risk?

Is the Technology Maturation and Risk Reduction (TMRR) Development work effort properly staffed?

Is the Technology Maturation and Risk Reduction (TMRR) Development work effort executable within the existing budget and schedule?

Has a preliminary system specification, consistent with technology maturity and the proposed program cost and schedule, been captured in the system technical baseline?

Supportability: In addition to asking whether the system is supportable in the field, you should also ask:

Have the preliminary manufacturing processes and risks been identified for prototypes?

Have initial producibility assessments of design concepts been completed?

Is the program schedule for the Technology Maturation and Risk Reduction (TMRR) Development Phase executable (technical/cost risks)?

Are the hazards sufficiently understood and addressed to achieve an acceptable/manageable level of ESOH risk in the Technology Maturation and Risk Reduction (TMRR) Development phase?

(Source: DAG P. 230)

System Functional Review

Affordability: In addition to answering the question about the existing budget of the system, you should also ask:

Does the updated cost estimate fit within the existing budget?

Are adequate processes and metrics in place for the program to succeed?

Are the risks known and manageable for development?

Is the program schedule executable (technical/cost risks)?

Is the updated Cost Analysis Requirements Description (CARD) consistent with the approved functional baseline?

Functionality: In addition to asking questions about whether system design can proceed you should also ask:

Has the system Functional Baseline been established to enable preliminary design to proceed with proper configuration management?

Can the system functional requirements, as disclosed, satisfy the draft Capability Development Document?

Has a draft preliminary Software Requirements Specification been defined, with complete verification requirements?

Has an initial software architecture design been defined?

Is the software functionality in the approved functional baseline consistent with the updated software metrics and resource-loaded schedule?

Supportability: In addition to asking questions about whether Supportability requirements are included in specifications you should also ask:

Are the program development efforts required to achieve the sustainment key performance parameters, key system attributes, and enabler metrics, along with their corresponding schedules, included in the program documentation and Life-cycle Sustainment Plan?

Have the system-level hazards been reviewed and mitigating courses of action been identified?

Is the program properly staffed?

(DAG, pp. 243-5)

Preliminary Design Review

Affordability: In addition to asking questions about the existing budget you should also ask:

Are the risks known and manageable for integrated testing and developmental and operational evaluation?

Is the program schedule executable (technical/cost risks)?


Has the program‘s cost estimate been updated?

Is the preliminary system level design producible within the production budget?

Is the updated CARD consistent with the approved allocated baseline?

Design Readiness: In addition to asking questions about manufacturing processes you should also ask:

Does it follow functional review requirements?

Does the status of the technical effort and design indicate operational test and evaluation success (operationally effective and suitable)?

Can the preliminary design, as disclosed, satisfy the draft Capability Development Document?

Has the system allocated baseline been established and documented to enable detailed design to proceed with proper configuration management?

Have the majority of manufacturing processes been defined and characterized?

Are initial manufacturing approaches documented?

Have producibility assessments of key technologies been completed?

Has a production cost model been constructed?

Supportability: In addition to asking questions about meeting logistics requirements you should also ask:


Have sustainment and human integration design factors been reviewed and included, where needed, in the overall system design?

Can the industrial base support production of development articles?

Have long-lead and key supply chain elements been identified?

Can the risks associated with ESOH hazards be mitigated to an acceptable risk level within the existing budget?

(DAG, pp. 247-50)

Critical Design Review

Affordability: In addition to asking questions about the existing budget you should also ask:

Is the detailed design producible within the production budget?

Is the updated Cost Analysis Requirements Description (CARD) consistent with the approved product baseline?

Does the updated cost estimate fit within the existing budget?

Design Capability: In addition to asking questions about the design satisfying the Capabilities Development Document you should also ask:

Does the status of the technical effort and design indicate operational test and evaluation success (operationally effective and suitable)?

Has the system product baseline been established and documented to enable hardware fabrication and software coding to proceed with proper configuration management?

Is the software functionality in the approved product baseline consistent with the updated software metrics and resource-loaded schedule?

Have key product characteristics having the most impact on system performance, assembly, cost, Reliability, and sustainment or safety been identified?

Have the critical manufacturing processes that affect the key characteristics been identified and their capability to meet design tolerances determined?

Have process control plans been developed for critical manufacturing processes?

Have manufacturing processes been demonstrated in a production representative environment?

Are detailed trade studies and system producibility assessments underway?

Are materials and tooling available to meet pilot line schedule?

Has the system production cost model been updated, allocated to subsystem level, and tracked against targets?

Are long lead procurement plans in place and has the supply chain been assessed?

Are the ESOH residual risks known and manageable?

Supportability: In addition to asking questions about meeting logistics requirements you should also ask:

Supportability requirement enablers, such as Human Systems Integration (HSI) design features, inclusive of the environment, safety and occupational health risk reduction features, are included in the design.

Failure Mode Effects and Criticality Analysis have been completed and any remaining subsystem requirements for the product support package elements design are complete.

Key sustainment characteristic drivers (including critical manufacturing processes to achieve them) have been identified, and an estimate of system Reliability based on demonstrated Reliability rates and other sustainment drivers are being used in developing the product support package.

Development testing results are used to update the sustainment metric projection estimates and any planned corrective actions to hardware/software deficiencies have been identified.

Test success criteria for any remaining development testing and operational testing plans (for testing both operationally effective and suitable) for key sustainment enablers or drivers requirements are complete. If the test results to date do not indicate the operational test success is likely or risk has increased, new developmental and operational testing criteria and plans should be considered, along with fallback plans.

Program sustainment development efforts with corresponding schedules, including system fabrication, test, and software critical path drivers, are included in LCSP updates.

Has the detailed design satisfied sustainment and Human Systems Integration requirements?


Are the risks known and manageable for testing in support of developmental and operational evaluation objectives?

(DAG, pp. 262-5)

Developmental Test and Evaluation

Are critical system design requirements, including Supportability metrics, being met?

Is there a negative or positive influence on Supportability from the Reliability predictions and assessments based on the test data

(DAG, pp. 802-3)

Functional Configuration Audit

Functionality: Do all the subsystems do what they are supposed to do according to the functional configuration?

(DAG, pp. 267-8)

Production Readiness Review

Additional Producibility questions you should consider are:

Do we have long lead items?

Are all technologies mature enough for production?

Is the supply chain established and stable with materials available to meet planned low rate production?

Have we identified the skills and machines?


Are the production facilities ready and required workers trained?

Are the ESOH residual risks known and manageable?

Have we laid out processes and process controls?

Has the system product baseline been established and documented to enable hardware fabrication and software coding to proceed with proper configuration management?


Are the risks known and manageable?

Is the detailed design producible within the production budget?

(DAG, pp. 268-9)

42 of 58

January 2013

Final v1.3

January 2013

Final v1.3

43 of 58