Test Planning and Analysis

EMIS 7307

T&E Part 5

1

Test Planning and Analysis

Landing distance of an F-18

EMIS 7307

T&E Part 5

2


EMIS 7307

T&E Part 5

3


EMIS 7307

T&E Part 5

4

• What if 60% isn’t sufficient confidence?– Typically 95% is the desired confidence.

• Need more data points!– How many?– Can’t say for sure.– Depends on s and x with the new data.


EMIS 7307

T&E Part 5

5


Let’s assume that if we collect more data the standard deviation and (x -V*) don’t change. This not a very good assumption, but possibly the best we can do. Then we can rewrite ts as ts= k *√(n-1).

With this assumption k = .351.

For:

n =2, ts= .351, α=.39, (1-α) = .61

n =5, ts= .702, α=.27, (1-α) = .73

n =10, ts= 1.053, α=.16 (1-α) = .84

n = 17, ts = 1.404, α=.09 (1-α) = .91

n = 26, ts = 1.755, α=.04 (1-α) = .96

n = 37, ts = 2.106, α=.015 (1-α) = .985

EMIS 7307

T&E Part 5

6


•From the previous chart we get a sense of the “cost” of confidence.

•For an increase of only three tests (from 2 to 5) we can boost our confidence by 12%.

•But the change from 17 to 26 only further increased our confidence by 5%!

•This illustrates why 90% confidence is often chosen as good enough. It’s a compromise of increased confidence versus cost.

EMIS 7307

T&E Part 5

7


ANOVA

R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers andScientists, Prentice-Hall, Englewood Cliffs, New Jersey.

EMIS 7307

T&E Part 5

8



EMIS 7307

T&E Part 5

9


ANOVA


EMIS 7307

T&E Part 5

10



EMIS 7307

T&E Part 5

11



EMIS 7307

T&E Part 5

12



EMIS 7307

T&E Part 5

13



EMIS 7307

T&E Part 5

14



EMIS 7307

T&E Part 5

15


•In this example the table f value is 2.78. Use 4 and 25 as the degrees of freedom.

•The calculated f is 4.3.

•Therefore, reject the null hypothesis.


EMIS 7307

T&E Part 5

16


EMIS 7307

T&E Part 5

17


• ANOVA is a powerful data analysis tool– Using a few samples answers the question, does

an effect on a subgroup of a hypothetical population cause that subgroup to not really be from the hypothetical population?

• We hypothesized that the five different aggregates were from a population that absorbed moisture similarly.

• The ANOVA showed that they were not!

EMIS 7307

T&E Part 5

18


• ANOVA is readily available in many spreadsheet packages.– Excel has it under tools then click on data

analysis and select ANOVA.

EMIS 7307

T&E Part 5

19


• Bottom line.– Use student t or normal distribution (n>30)

when comparing to a spec.– Use ANOVA when looking to see if two

groups are statistically the same to some confidence level.

EMIS 7307

T&E Part 5

20


• Bottom line (cont’d)– For both test planning and data analysis at least

one member of the team needs to be statistics savvy.

– From a management perspective.• This facilitates planning for adequate data collection

to meet the desired confidence. • Assures data collection ends when sufficient data

exist.

EMIS 7307

T&E Part 5

21

Special TopicSoftware testing

• In an earlier lecture I presented the results of a study showing the economic impact of poor or limited software testing.

• I’ve also made the case that earlier problem finding and fixing is exponentially cheaper.

• Here I want to introduce several software testing concepts.

EMIS 7307

T&E Part 5

22

Percent of Specification Requirements Involving Software

0

20

40

60

80

100

1960 1964 1970 1975 1982 1990 2000

Percent

Series1

F-4

A-7

F-111

F-15 F-16

B-2

F-22

Ref: Lockheed - Martin Corp.

EMIS 7307

T&E Part 5

23


•Human testing is defined as informal, noncomputer-based methods of evaluating architectures, designs and interfaces. It can consist of:

• Inspections – The programmer explains his/her work to a small group of peers with discussion and direct feedback on errors, inconsistencies and omissions.• Walk-through – A group of peers develop test cases to evaluate work to date and give direct feedback to the programmer.• Desk Checking – A self evaluation is made by the programmer of his/her own work. There is a low probability of identifying his/her errors of logic or coding.

EMIS 7307

T&E Part 5

24


• Peer Ratings – Mutually supportive, anonymous reviews are performed by groups of peers with collaborative evaluations and feedback.• Design Reviews – Preliminary design reviews (PDRs) and critical design reviews (CDRs) provide milestones in the development efforts that review development and evaluations to date.

EMIS 7307

T&E Part 5

25


• Computerized software testing– Black box

• Functional testing of a software unit without knowledge of how the internal structure or logic will process the input to obtain the specified output. Within-boundary and out-of-boundary stimulants test the software’s ability to handle abnormal events. Most likely cases are tested to provide a reasonable assurance that the software will demonstrate specified performance. Even the simplest software designs rapidly exceed capacity to test all alternatives.

EMIS 7307

T&E Part 5

26


• Computerized software testing– White box

• Structural testing of the internal logic and software structure provides an opportunity for more extensive identification and testing of critical paths. The process and objectives are otherwise very similar to black box testing.

EMIS 7307

T&E Part 5

27


• IV&V– Independent verification and validation are risk-

reducing techniques that are applied to major software development efforts. The primary purpose of IV&V is to ensure that software meets requirements and is reliable and maintainable. The IV&V is effective only if implemented early in the software development schedule. Requirements analysis and risk assessment are the most critical activities performed by IV&V organizations.

EMIS 7307

T&E Part 5

28


• IV&V– The desire to boast of a high SEI rating in proposals

and to generally improve the corporations efficiency has moved many companies to significantly improved software development capabilities.

– This in turn has reduced the perceived need by customers to require and pay for IV&V.

Documents

Test Planning and Analysis