Upload
jocelyn-allison
View
223
Download
2
Tags:
Embed Size (px)
Citation preview
Chin-Yu HuangDepartment of Computer Science
National Tsing Hua UniversityHsinchu, Taiwan
Optimal Allocation of Testing-Resource Considering Cost, Reliability, and Testing-
Effort
The 10th International Symposium Pacific Rim Dependable Computing
(PRDC 2004)
2
Outline Introduction Software Reliability Growth Model with
Generalized Logistic Testing-Effort Function Testing Resource Allocation for Software
Module Testing Experimental Studies and Results —Numerical Example and Sensitivity
Analysis Conclusions
3
Introduction A software testing process consists of several
testing stages including module testing, integration testing, system testing, and installation testing.
The quality of the tests usually corresponds to the maturity of the software test process, which in turn relates to the maturity of the overall software development process.
Most popular and commercial software products are complex systems and composed of a number of modules.
4
Introduction (contd.)
Project managers should know how to allocate the specified testing-resources among all the modules and develop quality software with high reliability.
We provide a systematic method for the project managers to allocate specific amount of testing-resource expenditures for each module under some constraints: minimizing the cost of software development, with a given fixed amount of testing-effort and a reliability objective.
5
SRGMs Software Reliability represents a customer-oriente
d view of software quality and is dynamic rather than static.
SRGMs describe failures as a random process, which is characterized in either times of failures or the number of failures at fixed time.
We use an SRGM with generalized logistic testing-effort function to describe the time-dependency behaviors of detected faults and the testing-resource expenditures spent during module testing.
6
Generalized Logistic Testing-Effort Function The generalized logistic testing-effort function (LTEF) was proposed by Huang, Kuo, and Lyu (1999) an
d can be depicted as
where N is the total amount of testing effort to be eventually consumed, a is the consumption rate of testing-effort expenditures, A is a constant, and k is a structuring index with a large value for modeling well-structured software development efforts.
tAe
NtW
1
7
SRGM with Generalized Logistic Testing-Effort Function
))])()((exp[()( 0WtWr1atm ))]((exp[ tWr1a
where m(t) is the expected mean number of faults detected in time (0, t) , Wk(t) is the current testing-effort consumption at time t, a is the expected number of initial faults, and r is the error detection rate per unit testing-effort at testing time t & r>0.
This SRGM with Generalized LTEF was proposed by Huang, Kuo, and Lyu (1999) and the mean value function is
8
SRGM with Generalized Logistic Testing-Effort Function (contd.)
When t→∞, the expected number of faults to be detected is
The reliability of a software system is defined as the ratio of the cumulative number of detected faults at time t to the expected number of initial faults:
aA
Nam
]
1exp[1)( (if A >> N)
))(exp(1)(
)( trWa
tmtR
9
Testing-Resource Allocation for Module Testing
Assumptions :
1) The software system is composed of N independent software modules that are tested individually. The number of faults remaining in each module can be estimated by an SRGM with generalized LTEF.
2) For each module, the failure data have been collected and the parameters of each module can be estimated.
3) The total amount of testing resource expenditures available for the module testing processes is fixed and denoted by W.
10
Testing-Resource Allocation for Module Testing
Assumptions (contd.):
4) If any of the modules is faulty, the whole software system fails.
5) The system manager has to allocate the total testing resources W to each software module to minimize the number of faults remaining in the system during the testing period.
6) The desired software reliability after the testing phase is greater than or equal to the reliability objective R0.
11
Mean Value Function of a Software System with N Modules
The number of detected faults in the system can be estimated by
where vi is a weighting factor to measure the relative importance of a fault removal from module i in the future.
If vi =1 for all i =1, 2,…, N, the objective is to minimize the total number of faults remaining in the software system after the testing phase.
N
iiiii
N
iii tWravtmv
11)))(exp(1()(
12
Mean Value Function of a Software System with N Modules (contd.)
Therefore, the number of remaining faults can be estimated by
N
iiiii
N
iii WravtWrav ii
11)exp())(exp(
13
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
The objective function is : minimizing the cost of software testing
Subject to the constraints : (1) The total amount of testing resource expenditures for all modules would not be more than available resource W. (2) Reliability of each module should not be less than R0 guarantee that reliability of system will not be less than R0 .
14
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
The objective function is:
Minimize:
Subject to the constrains:
, i=1, 2,..., N.
)(1
N
iii WCost
0,1
i
N
ii WWW
0))(exp(1)( RtWrtR iii
15
Therefore, we have:
, i= 1,2,…,N )1ln(1
0Rr
Wi
i
)1ln(1
0Rr
Di
i
Let
),.....,,,,0max( 321 NDDDDC
We can have , ,i=1,2,….,N
N
ii WW
1
0iW
and ,whereCWi
Let , we can transform the above equations to
The following two equation:
CWX ii
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
16
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
Minimize:
Subject to
)(*1
N
iii XCost
0,11
i
N
ii
N
ii XCWX ,i=1,2, ..., N.
where
N
iiiii
N
iii CravCXCost
11 )exp(1(*)(*
1
N
iiiiiiiii XrCravCXr
12 )exp()exp(*))exp(
N
iii CXC
13 )(*
17
According to the Lagrange multiplier, above equationsbe simplified as follows:
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
Minimize:
N
iiiiiiiN XrCravCXXXL
11 ))exp()exp(1(*),,...,,( 21
)exp()exp(*1
2 iiiiii XrCravCN
i
)()(*111
3
N
ii
N
iiii CWXCXC
N
i
18
Based on the Kuhn-Tucker conditions (KTC) , the necessary conditions for a minimum value of above equation are in existence .
The results will be applied in the Algorithm for solving our problem.
Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective
19
Algorithm 1Step 1: Set l=0.
Step 2: Calculate the following equations
)}*ln())exp()**({ln(1
312 CCrCCravr
X iiiiii
i ,i=1, 2,..., N- l.
3*C
N
ii
N
i
N
iiiiiiii
r
CWCrCCravr
1
1 1
)/1(
))exp()**()(ln/1(exp
12
Step 3: Rearrange the index i such that .... ***21 lNXXX
Step 4: IF then0* lNX stop (i.e., the solution is optimal)0* lNXElse ; l=l+1
End-IF.Step 5: Go to Step 2.
20
Observation 1
The optimal solution has the following form :
otherwiseX
r
CWCrCCravr
Cwhere
lNi
CCrCCravr
X
i
N
ii
N
i
N
iiiiiiii
iiiiii
i
,0
)/1(
))exp()**()(ln/1(exp
*
.,...,2,1,
)}*ln())exp()**({ln(1
*1
1 1
*
12
3
312
21
Observation 2 Algorithm 1 always converges in, at worst, N-1 steps and thus the
value of objective function at the optimal solution is
),...,,( **2
*1 NXXX
N
iiiiiii CXrCravC
12
*1 *))exp()exp(1(*
N
ii
N
iiiiiii CXCXrCrav
1
*13
1
* )(*)exp()exp(
22
Numerical Examples Suppose the total amount of testing-effort expendit
ures W is given.
We apply the proposed model to actual software failure data and have to allocate the expenditures to each module to minimize the expected cost.
All the parameters ai and ri for each software module have been estimated by using the maximum likelihood estimation (MLE) or the least squares estimation (LSE).
23
Table 1 : the estimated values of ai , ri , vi , and .Module ai ri vi
1 89 4.1823×10-4 1 1.0 7632
2 25 5.0923×10-4 1 0.6 3158
3 27 3.9611×10-4 1 0.7 4009
4 45 2.2956×10-4 1 0.4 4329
5 39 2.5336×10-4 1 1.5 8964
6 39 1.7246×10-4 1 0.5 4568
7 59 8.819×10-5 1 0.5 6023
8 68 7.274×10-5 1 0.6 9112
9 37 6.824×10-5 1 0.05 0
10 14 1.5309×10-4 1 1 2203
*iX
An example for Proposed Algorithm
24
Sensitivity analysis
If a1 is increased by 40%, then the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 8400 and its relative change (RC) is 0.100628931 about 10% increment).
for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 0.95%, 0.95%, 1.52%, 0.67%, 1.93%, 2.87%, 2.30%, and 4.49%, respectively.
25
Sensitivity analysis (contd.)
1 2 3 4 5 6 7 8 9 10
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10
Re
lativ
e C
ha
ng
e
Modules
a1*1.4
a1*1.3
a1*1.2
a1*1.1
Figure 1: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a1.
26
If a1 is decreased by 30% the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 6818 and its RC is 0.106656184 (about 10.66% decrement). for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are increased by about 1.01%, 1.02%, 1.64%, 0.71%, 2.06%, 3.05%, 2.44%, and 4.86%, respectively.
Sensitivity analysis (contd.)
27
Sensitivity analysis (contd.)
1 2 3 4 5 6 7 8 9 10
-0.16
-0.14
-0.12
-0.10
-0.08
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10R
ela
tive
Ch
an
ge
Modules
a1*0.6
a1*0.7
a1*0.8
a1*0.9
Figure 2: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% decrease to a1.
28
Sensitivity analysis (contd.)
If a1 & a2 both are increased by 40%
the estimated values of optimal testing-effort expenditure for modules 1 and 2 are changed from 7632 to 8370 (about 9.67% increment) and 3158 to 3764 (about 19.18% increment), respectively.
for modules 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 1.75%, 2.79%, 1.23%, 3.52%, 5.25%, 4.19%, and 8.22%, respectively.
29
Sensitivity analysis (contd.)
Figure 3: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a1 & a2.
1 2 3 4 5 6 7 8 9 10
-0.10
-0.08
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.18
0.20
0.22R
ela
tive
Ch
an
ge
Modules
a1*1.4 & a
2*1.4
a1*1.3 & a
2*1.3
a1*1.2 & a
2*1.2
a1*1.1 & a
2*1.1
30
Conclusions In this paper, we proposes a method to optimize
the software testing-resource allocation problem —minimizes the cost of software development, with a given fixed amount of testing-effort and a reliability objective.
We develop a comprehensive strategy for module testing in order to help software project managers make the best decisions in practice.
31
Conclusions (contd.)
An extensive sensitivity analysis is presented to study the effects of various principal parameters on the optimization problem of testing-resource allocation.
Using Algorithm 1, project managers can allocate limited testing-resource easily and efficiently and thus achieve the lowest cost objective during software module and integration testing.