View
231
Download
0
Category
Preview:
Citation preview
6. Verify and sell-off 1
Agenda for Verify Activity
1. Objectives 2. Verification Vs validation 3. Verification methods 4. Documentation of tests 5. Test execution 6. Test results 7. Reviews 8. Special tests
6. Verify and sell-off 2
1. Objectives
Verify activity Product-based activities Products used to manage Completion criteria
1. Objectives
6. Verify and sell-off 3
Verify Activity
Verifies that the product complies with all its requirements, and documents these test results
1. Objectives
6. Verify and sell-off 4
Verify Tasks
Write test plan
Write test spec
Write test procedure
product spec
Review test
readiness
Verifyproduct
by analysis
Review verification
results
Verifyproduct by ins, demo,
test, sta
test results
test plan
product
test spec
test proc
insdemoteststa
ana
product design
1. Objectives
6. Verify and sell-off 6
Pseudo- Completion Criteria
Testing complete Product verification review complete
1. Objectives
6. Verify and sell-off 7
2. Verification Vs Validation
Verification Validation Heuristic Government Vs civilian validation
2. Verification vs validation
6. Verify and sell-off 8
Verification
Formal collection of test results that show compliance to requirements
Can we prove that the product meets the requirements of the specification
2. Verification vs validation
6. Verify and sell-off 9
Validation
Assessment of the correctness of a product for its end-use.
Can we show that the requirements reflect the needs of the end user
2. Verification vs validation
6. Verify and sell-off 10
Heuristic
Verification -- Have we built the system right Validation -- Have we built the right system
2. Verification vs validation
6. Verify and sell-off 11
Government Vs Civilian Validation
Government -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL)
Civilian -- Contractor often validates needs
2. Verification vs validation
6. Verify and sell-off 12
3. Verification Methods
Inspection Analysis Demonstration Test
3. Verification methods
6. Verify and sell-off 13
Inspection (INS)
Verification without the use of special laboratory appliances or procedures.
Generally non-destructive. Includes sight, hearing, smell, touch, and taste May include simple physical manipulation May include mechanical & electrical measurement May include other forms of investigation
Done when convenient, normally when no further changes to the configuration are planned
3. Verification methods
6. Verify and sell-off 14
Analysis (ANA)
Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals
Verification through the technical evaluation of data
Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation
MTBF and MTBCF analysis may include failures from testing
3. Verification methods
6. Verify and sell-off 15
Analysis Categories
Math analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements
Design analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements
Extrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements
Analysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests.
3. Verification methods
6. Verify and sell-off 16
Demonstration (DEM)
Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.
3. Verification methods
6. Verify and sell-off 17
Test (TST)
Verification by technical means, including operation, and involves application of scientific principles and procedures.
3. Verification methods
6. Verify and sell-off 18
Selected Test and Analysis (STA)
Verifies a quantitative requirement by selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program
Conducted under controlled conditions Analysis needed to interpret test results and to
extend test results to points not tested Models need to be validated
3. Verification methods
6. Verify and sell-off 19
Distinctions of Methods
Subjective Agreement between customer &contractor
3. Verification methods
6. Verify and sell-off 20
Crude Distinction
Analysis -- Verification that can occur without the product being present
Inspection -- Simple verification without exercising the product
Demonstration -- Simple verification while exercising the product
Test -- Everything else
3. Verification methods
6. Verify and sell-off 21
Test
Many opinions say that test involves exercising the product and collecting data that needs to be reduced
3. Verification methods
6. Verify and sell-off 22
Importance of Distinguishing among Methods
Distinction is not terribly important Analysis represents less risk Inspection and demonstration are easier than test Many methods change when test spec written
3. Verification methods
6. Verify and sell-off 23
Analysis Examples
Test coverage is greater than 95 percent Design conforms to best commercial practices
3. Verification methods
6. Verify and sell-off 24
Inspection Examples
Weighs less than 10 pounds Surface free of visible defects
3. Verification methods
6. Verify and sell-off 25
Demonstration Examples
Closed caption text appears when commanded Light goes off when safety switch activated
3. Verification methods
6. Verify and sell-off 26
Test Examples
Signal to noise ratio > 2 Vibration power spectral density <0.2 g2/Hz at 20 Hz
3. Verification methods
6. Verify and sell-off 27
4. Documentation of Tests
Test plan Test specification Test procedure
4. Documentation of tests
6. Verify and sell-off 28
Test Plan (1 of 2)
Generated by the contractor Defines approach to testing Establishes approach to test methods
Demonstration, test, & STA -- defined in test spec Analysis -- defined in verification by analysis plan Inspection -- defined as inspection directives
4. Documentation of tests
6. Verify and sell-off 29
Test Plan (2 of 2)
Defines roles of test specs and test procedures Defines how to capture test results Defines reviews
4. Documentation of tests
6. Verify and sell-off 30
Test and Evaluation Master Plan
Generated by customer Defines customer plan for evaluating product Applies to the highest level government product
4. Documentation of tests
6. Verify and sell-off 31
Test Specification
Generated by contractor Defines agreement between customer and
contractor about what each demonstration, test, and STA must do
Includes the pass/fail criteria Is not a substitute for the product specification
4. Documentation of tests
6. Verify and sell-off 32
Test Specification Outline
1.0 Purpose -- gives high level purpose 2.0 Scope -- covers DEM, TST, STA test methods 3.0 Objectives -- states to verify requirements 4.0 Document organization -- describes each section 5.0 Overview -- describes test approach and strategy 6.0 Detailed tests -- defines tests - 7.0 Verification map -- assigns requirements to tests 8.0 Applicable documents -- lists cited documents 9.0 Acronyms -- lists acronyms used
4. Documentation of tests
6. Verify and sell-off 33
Detailed Test Group Outline (1 of 2)
1. Test objectives -- requirements verified by test group 2. Test prerequisites -- tests or activities that must be
completed prior to this test 3. Test requirements -- what is required to execute the
test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions
4. Documentation of tests
6. Verify and sell-off 34
Detailed Test Outline (2 of 2)
4. Test conditions -- test conditions and ordering 5. Analysis requirements -- analysis techniques
and tools needed to determine pass/fail 6. Pass/fail criteria -- exit criteria for test
DEM -- what should be observed TST, STA -- what should be measured ALL -- margin of error, tolerance
4. Documentation of tests
6. Verify and sell-off 35
Analysis Plan
Generated by contractor Defines agreement between customer and
contractor about what each analysis must do Includes the pass/fail criteria Is not a substitute for the product specification
4. Documentation of tests
6. Verify and sell-off 36
Inspection Directive
Generated by contractor Defines agreement between customer and
contractor about what each inspection must do Includes the pass/fail criteria Usually a single sheet that also documents
results of test Is not a substitute for the product specification
4. Documentation of tests
6. Verify and sell-off 37
Test Description
Requirements Design Build
Test spec Test description Test procedure
Expands detail between the test spec & procedures Analogous to design Allows people to agree on details of test approach
before starting to write test procedure
is similar to
4. Documentation of tests
6. Verify and sell-off 38
Test Procedure
Generated by the contractor Defines specific test set up Defines specific test process for running test Applies to demonstration, test, and selected test
and analysis
4. Documentation of tests
6. Verify and sell-off 39
5. Test Execution
Simulation Configuration Tests Regression tests
5. Test execution
6. Verify and sell-off 40
Simulation
A test tool that reduces cost Verifies extremes that would be hard to produce
in actual tests Support s performance and environmental testing
5. Test execution
6. Verify and sell-off 41
Simulation Verification
Simulations need to be validated for accuracy Customer may not have confidence in simulation
if the phenomena being simulated is complicated; e.g. software throughput
5. Test execution
6. Verify and sell-off 42
Configuration
Tests should be done on configured hardware and software to ensure repeatability
5. Test execution
6. Verify and sell-off 43
Tests
May be run in a lab, in the manufacturing environment, when installed, or when used in the field
Important tests may warrant a quick-look report giving test results within 24 hours of the test
5. Test execution
6. Verify and sell-off 44
Regression Tests
Tests that are run to confirm that a product performance is OK after a change to the product
Run after originally scheduled verification test For example, if a new device driver is added to an
already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver
5. Test execution
6. Verify and sell-off 45
6. Test Results
Test method
Analysis (ANA)
Inspection (INS)
Demonstration (DEMO)
Test (TEST)
STA
Pass/fail
VDB
VDB
VDB
VDB
VDB
Analysis
Study
n/a
n/a
Study
Study
Test
Sheet
Sheet
Sheet
Report
Report
Pass/fail status for each type of test is recorded in a single place such as a verification data base (VDB), Analysis is documented in a study. Tests having methods ANA, INS, or DEMO are documented in a single sheet report. Test and STA are documented in a separate report
6. Test results
6. Verify and sell-off 46
7. Reviews
Readiness review Product verification review Failure review board
7. Reviews
6. Verify and sell-off 47
Readiness Review
Obtains customer concurrence that contractor is ready to perform test
Confirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test
Presents schedule of testing, analysis, and review
7. Reviews
6. Verify and sell-off 48
Product Verification Review (1 of 2)
Obtains customer concurrence that testing is satisfactory
Held after test Examines results of tests, including analysis
7. Reviews
6. Verify and sell-off 49
Product Verification Review (2 of 2)
List tests that pass Explains exceptions, problems, and failures;
addresses deviations and waivers May be executed as a series of reviews with each
review associated with one or more tests
7. Reviews
6. Verify and sell-off 50
Failure Review Board
Examines each hardware failure Defines failure mechanism such as electrostatic
discharge, manufacturing defect, or manufacturing Defines impact of failure Defines options for preventing future failures
7. Reviews
6. Verify and sell-off 51
8. Special Tests
Reliability testing Design of experiments Accelerated life testing Maintainability and testability Environmental stress screening (ESS) Electromagnetic testing Failure report and corrective action system
8. Special tests
6. Verify and sell-off 52
Agenda for Sell Off Activity
1. Objectives 2. Sell off Vs close out 3. Audits 4. Importance of sell off 5. Homework
6. Verify and sell-off 53
1. Objectives
Sell-off activity Product-based activities Products used to manage Completion criteria
1. Objectives
6. Verify and sell-off 54
Sell-Off Activity
Confirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product
1. Objectives
6. Verify and sell-off 55
Sell-Off Tasks
ConductPCA
ConductFCA
Collect SOW deliverables and
test results
Agree
product
FCA audit results
SOW deliverables and test results
SOW deliverables and test results
agreement
PCA audit results
1. Objectives
6. Verify and sell-off 57
Pseudo-Completion Criteria
Product delivered FCA and PCA complete
1. Objectives
6. Verify and sell-off 58
2. Sell Off Vs Close Out
Sell off occurs when the product has completed verification
Contract close out comes later when all contractual issues are closed
2. Sell off vs close out
6. Verify and sell-off 59
3. Audits
Surveys to confirm delivered product is as it should be Major government audits
Functional configuration audit (FCA) Physical configuration audit (PCA)
May alternately be considered part of verification
3. Audits
6. Verify and sell-off 60
FCA
Verifies that the each hardware and software configuration item complies with its requirements
Confirms that product passed its tests Example items in audit
Test procedures agree with spec & test spec Test results Configuration control of testing Action items
3. Audits
6. Verify and sell-off 61
PCA
Establishes the product baseline, ensures as-built product matches documentation, verify that acceptance testing is adequate
Confirms we built what we designed Example items in audit
CI design documentation History of changes made and pending Operating and support manuals Parts selection list CI as-built specifications and interfaces Manufacturing and assembly instructions Acceptance test procedures and data
3. Audits
6. Verify and sell-off 62
Types of PCAs
Build up -- Perform PCA as product built up Tear down - Perform PCA after product built
3. Audits
6. Verify and sell-off 63
4. Importance of Sell Off
Important for completing contract Not a difficult concept Requires planning to assemble documentation
Suggestions Make the customer a team member in sell off
so that the customer will know what to expect at the sell off
Begin at CDR to monitor the percentage of all sell-off items that are complete
4. Importance of sell off
6. Verify and sell-off 64
Example 1 -- Government Product
Contract items Product Test results Warranty Spare parts Support agreement Government furnished equipment Facilities Contract data
Sell-off agreement DD 250 for each contract line item
4. Importance of sell off
6. Verify and sell-off 65
Example 2 -- House
Sell-off items House Title policy Warranty deed
Sell-off agreement Bill of sale
4. Importance of sell off
6. Verify and sell-off 66
5. Homework
1. For buying a car for personal use, list three types of tests that might apply for each of the four test methods.
2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use?
3. In building a house, what is an example of a verification test that we might require before the house is complete?
5. Homework
Recommended