9. Verify and sell-off1 Agenda for verify activity r1. Objective r2. Verification Vs validation r3. Verification methods r4. Test documentation r5. Test

9. Verify and sell-off 1

Agenda for verify activity1. Objective2. Verification Vs validation3. Verification methods4. Test documentation5. Test execution6. Test results7. Reviews8. Special tests


1. ObjectiveVerify activityVerify tasksCompletion criteriaPseudo-completion criteria

1. Objective


Verify activityVerifies that the product complies with all its

requirements, and documents these test results

1. Objective


Verify tasks

Write test plan

Write test spec

Write test procedure

product spec

Review test

readiness

Verifyproduct

by analysis

Review verification

results

Verifyproduct by ins, demo,

test, sta

test results

test plan

product

test spec

test proc

insdemoteststa

ana

product design

1. Objective


Completion criteriaCustomer agrees testing is complete

1. Objective


Pseudo-completion criteriaTesting completeProduct verification review complete

1. Objective


2. Verification vs validationVerificationValidationHeuristicGovernment vs civilian validation

2. Verification vs validation


VerificationFormal collection of test results that show

compliance to requirementsCan we prove that the product meets the

requirements of the specification



ValidationAssessment of the correctness of a product

for its end-use.Can we show that the requirements reflect

the needs of the end user



HeuristicVerification -- Have we built the system rightValidation -- Have we built the right system



Government vs civilian validation

Government -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL)

Civilian -- Contractor often validates needs



3. Verification methodsInspectionAnalysisDemonstrationTestSelected test and analysisDistinction among methods

3. Verification methods


Inspection (INS) (1 of 2)

Definition: Verification without the use of special laboratory appliances or procedures. • Generally non-destructive. • Includes sight, hearing, smell, touch, and taste • May include simple physical manipulation• May include mechanical & electrical

measurement• May include other forms of investigation



Inspection (INS) (2 of 2)Done when convenient, normally when no

further changes to the configuration are planned

Examples• Weighs less than 10 pounds• Surface free of visible defects



Analysis (ANA) (1 of 3)Definition: Verification by technical or

mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals • Verification through the technical

evaluation of data• Data may may include lower-level product

test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation

• MTBF and MTBCF analysis may include failures from testing



Analysis (ANA) (2 of 3)Subtypes

• Math analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements

• Design analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements

• Extrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements



Analysis (ANA) (3 of 3)

• Analysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests.

Examples• Test coverage is greater than 95 percent• Design conforms to best commercial

practices



Demonstration (DEM)Definition: Verification using actual operation,

adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.

Examples• Closed caption text appears when

commanded• Light goes off when safety switch activated



Test (TST)Definition: Verification by technical means,

including operation, and involves application of scientific principles and procedures.

Many opinions say that test involves exercising the product and collecting data that needs to be reduced

Test examples• Signal to noise ratio > 2• Vibration power spectral density <0.2 g2/Hz at 20

Hz



Selected test and analysis (STA)Definition: Verifies a quantitative requirement by

selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program

Conducted under controlled conditionsAnalysis needed to interpret test results and to

extend test results to points not testedModels need to be validatedSTA example

• Performance shall be met at all altitudes from 1000 meters to 5000 meters



Distinctions among methods (1 of 2)SubjectiveAgreement between customer &contractorCrude distinction

• Analysis -- Verification that can occur without the product being present

• Inspection -- Simple verification without exercising the product

• Demonstration -- Simple verification while exercising the product

• Test -- Everything else



Distinctions among methods (2 of 2) Importance of distinguishing among methods

• Distinction is not terribly important• Analysis represents less risk • Inspection and demonstration are easier

than test• Many methods change when test spec

written



4. Test documentationTest planTest specificationTest procedure

4. Test documentation


Test Plan (1 of 3)Generated by the contractorDefines approach to testingEstablishes approach to test methods

• Demonstration, test, & STA -- defined in test spec

• Analysis -- defined in verification by analysis plan

• Inspection -- defined as inspection directives



Test Plan (2 of 3)Defines roles of test specs and test

proceduresDefines how to capture test resultsDefines reviews



Test plan (3 of 3)Test and evaluation master plan (TEMP

• Special case of test plan• Generated by customer• Defines customer plan for evaluating product• Applies to the highest level government

product



Test specification (1 of 6)Generated by contractorDefines agreement between customer and

contractor about what each demonstration, test, and STA must do

Includes the pass/fail criteriaIs not a substitute for the product specification



Test specification (2 of 6)Outline

• 1.0 Purpose -- gives high level purpose• 2.0 Scope -- covers DEM, TST, STA test

methods• 3.0 Objectives -- states to verify requirements• 4.0 Document organization -- describes each

section• 5.0 Overview -- describes test approach and

strategy• 6.0 Detailed tests -- defines tests -



Test specification (3 of 6)

• 7.0 Verification map -- assigns requirements to tests

• 8.0 Applicable documents -- lists cited documents

• 9.0 Acronyms -- lists acronyms used



Test specification (4 of 6)Detailed tests outline

• 1. Test objectives -- requirements verified by test group

• 2. Test prerequisites -- tests or activities that must be completed prior to this test

• 3. Test requirements -- what is required to execute the test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions



Test specification (5 of 6)

• 4. Test conditions -- test conditions and ordering

• 5. Analysis requirements -- analysis techniques and tools needed to determine pass/fail

• 6. Pass/fail criteria -- exit criteria for test• DEM -- what should be observed• TST, STA -- what should be measured• ALL -- margin of error, tolerance



Test specification (6 of 6)Analysis plan

• Generated by contractor• Defines agreement between customer and

contractor about what each analysis must do

• Includes the pass/fail criteria• Is not a substitute for the product

specification



Test procedure (1 of 3)Generated by the contractorDefines specific test set upDefines specific test process for running testApplies to demonstration, test, and selected

test and analysis



Test procedure (2 of 3)Inspection directive

• Generated by contractor• Defines agreement between customer and

contractor about what each inspection must do

• Includes the pass/fail criteria• Usually a single sheet that also documents

results of test• Is not a substitute for the product

specification



Test procedure (3 of 3)

Requirements Design Build

Test spec Test description Test procedure

Test description• Expands detail between the test spec & procedures• Analogous to design• Allows people to agree on details of test approach before starting

to write test procedure

is similar to



5. Test ExecutionSimulationConfigurationTestsRegression tests

5. Test execution


Simulation (1 of 2)A test tool that reduces costVerifies extremes that would be hard to

produce in actual testsSupport s performance and environmental

testing

5. Test execution


Simulation (2 of 2)Simulation verification

• Simulations need to be validated for accuracy

• Customer may not have confidence in simulation if the phenomena being simulated is complicated; e.g. software throughput

5. Test execution


ConfigurationTests should be done on configured hardware

and software to ensure repeatability

5. Test execution


TestsMay be run in a lab, in the manufacturing

environment, when installed, or when used in the field

Important tests may warrant a quick-look report giving test results within 24 hours of the test

5. Test execution


Regression testsDefinition: Tests that are run to confirm that a

product performance is OK after a change to the product

Run after originally scheduled verification testFor example, if a new device driver is added to

an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver

5. Test execution


6. Test Results

Test method

Analysis (ANA)

Inspection (INS)

Demonstration (DEMO)

Test (TEST)

STA

Pass/fail

VDB

VDB

VDB

VDB

VDB

Analysis

Study

n/a

n/a

Study

Study

Test

Sheet

Sheet

Sheet

Report

Report

Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports

provide supporting data

Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports

provide supporting data

6. Test results


7. ReviewsReadiness reviewProduct verification reviewFailure review board

7. Reviews


Readiness reviewObtains customer concurrence that contractor

is ready to perform testConfirms that test procedures are satisfactory,

that product to be tested will be configured, and that correct people will witness the test

Presents schedule of testing, analysis, and review

7. Reviews


Product verification review (1 of 2)Obtains customer concurrence that testing is

satisfactoryHeld after testExamines results of tests, including analysis

7. Reviews


Product verification review (2 of 2)List tests that passExplains exceptions, problems, and failures;

addresses deviations and waiversMay be executed as a series of reviews with

each review associated with one or more tests

7. Reviews


Failure review board

Examines each hardware failureDefines failure mechanism such as

electrostatic discharge, manufacturing defect, or manufacturing

Defines impact of failureDefines options for preventing future failures

7. Reviews


8. Special testsReliability testingDesign of experimentsAccelerated life testingMaintainability and testabilityEnvironmental stress screening (ESS)Electromagnetic testingFailure report and corrective action system

8. Special tests


Agenda for sell-off activity1. Objective2. Sell off vs close-out3. Audits4. Importance of sell-off5. Examples5. Homework


1. ObjectiveSell-off activitySell-off tasksCompletion criteriaPseudo-completion criteria

1. Objective


Sell-off activity

Confirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product

1. Objective


Sell-off tasks

ConductPCA

ConductFCA

Collect contract deliverables and

test results

Agree

product

FCA audit resultscontract

deliverables and test results

contract deliverables and

test results

agreement

PCA audit results

1. Objective


Completion criteriaSell off agreement complete

1. Objective


Pseudo-completion criteria

Product deliveredFCA and PCA complete

1. Objective


2. Sell-off vs close-out Sell off occurs when the product has completed

verificationContract close out comes later when all

contractual issues are closed

2. Sell-off vs close-out


3. Audits DefinitionFCAPCA

3. Audits


DefinitionSurveys to confirm delivered product is as it

should beMajor government audits

• Functional configuration audit (FCA)• Physical configuration audit (PCA)

May alternately be considered part of verification

3. Audits


FCAVerifies that the each hardware and software

configuration item complies with its requirements

Confirms that product passed its testsExample items in audit

• Test procedures agree with spec & test spec

• Test results• Configuration control of testing• Action items

3. Audits


PCA (1 of 2)

Establishes the product baseline, ensures as-built product matches documentation, verify that acceptance testing is adequate

Confirms we built what we designed Example items in audit

• CI design documentation• History of changes made and pending• Operating and support manuals• Parts selection list

3. Audits


PCA (2 of 2)

• CI as-built specifications and interfaces• Manufacturing and assembly instructions• Acceptance test procedures and data

Types of PCAs• Build up -- Perform PCA as product built up • Tear down - Perform PCA after product

built

3. Audits


4. Importance of sell-offImportant for completing contractNot a difficult conceptRequires planning to assemble

documentation Suggestions• Make the customer a team member in sell

off so that the customer will know what to expect at the sell off

• Begin at CDR to monitor the percentage of all sell-off items that are complete

4. Importance of sell off


5. ExamplesExample 1Example 2

5. Examples


Example 1Government product

• Contract items• Product

• Test results

• Warranty

• Spare parts

• Support agreement

• Government furnished equipment

• Facilities

• Contract data

• Sell-off agreement• DD 250 for each contract line item

5. Examples


Example 2House

• Sell-off items• House• Title policy• Warranty

deed

• Sell-off agreement

• Bill of sale

5. Examples


6. Homework1. For buying a car for personal use, list three

types of tests that might apply for each of the four test methods.

2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use?

3. In building a house, what is an example of a verification test that we might require before the house is complete?

6. Homework

Documents

9. Verify and sell-off1 Agenda for verify activity r1. Objective r2. Verification Vs validation r3. Verification methods r4. Test documentation r5. Test