Statistical Testing
Statistical Testing
Statistical testing randomly selects test cases from the input domain according to the intended usage distribution.
Statistical domain testing. Statistical finite state machine (or Markov
chain) testing. Statistical control flow testing. Statistical data flow testing.
Uses of Usage Distributions
Productivity improvement and schedule gains could be achieved by reducing testing effort for lesser-used parts.
New products can be introduced by implementing highly used features quickly to capture market share and then gradually adding lesser-used features in subsequent releases.
Uses of Usage Distributions
High return on investment: the cost-benefit ratio is reported to be about 1:10.
The use of usage distribution can foster better communications with target customers and help establish and maintain better customer relations.
Better Communications and Relations with Customers The construction of usage distribution needs
the direct or indirect involvement of customers. It is more likely for the software development organization and the customers to appreciate each other’s views and efforts in such a collaborative instead of adversarial environment.
Better Communications and Relations with Customers The use of usage distribution can help
develop more precise product requirements and specifications, which are more likely to satisfy customers’ needs.
Customer and user training can be better focused on those features that the customers use the most instead of esoteric ones more likely to be used by internal users.
Operational Profiles
Proportional domain testing uses operational profiles to maintain usage distributions.
An operational profile is a list of disjoint set of operations and their associated probabilities of occurrence.
It is a quantitative characterization of the way a software system is or will be used.
An ExampleFile type Hits % of total.gif 438536 57.47%.html 128869 16.89%directory 87067 11.41%.jpg 65876 8.63%.pdf 10784 1.41%.class 10055 1.32%.ps 2737 0.36%.ppt 2510 0.33%.css 2008 0.26%.txt 1597 0.21%.doc 1567 0.21%.c 1254 0.16%.ico 849 0.11%Cumulative 753709 98.78%Total 763021 100%
Usage frequencies(hits) and probabilities (% of total) for different file types for SMU/SEAS.
Representation of Operational Profiles It is customary to sort the operations by
descending probabilities of usage. It is common to have quite uneven
distribution of usage probabilities, with a few frequently used ones account for most of the usage frequencies.
Related to the uneven distribution of usage probabilities is the probability threshold for individual operations. If some operations have very low probability of usage, we could omit them in the operational profile.
Constructing Operational Profiles How many operational profiles are enough? This depends on the homogeneity or
similarity of product usage by different customers.
If there are no fundamental differences, one operational profile would be appropriate.
If substantial differences exist, then one operational profile for each individual group of customers would be appropriate.
Generic Methods
Actual measurement of usage at customer installations.
Survey of target customers. Usage estimation based on expert opinions.
Limitations of Actual Measurement For new products or new applications, there
won’t be in-field customer installations. Therefore, actual measurement would not be possible.
Even if there are in-field customer installations for products, the actual usage information and the environment may contain a substantial amount of business sensitive and proprietary data. Therefore, the customers would not be willing to share the data.
Expensive to implement.
Customer Surveys
Customer surveys can provide more objective and potentially more accurate information than expert opinions.
For products with limited number of customers, the survey might be more manageable.
For mass-market products, the massive user population requires careful planning to reduce the survey cost while maintaining result accuracy.
Expert Opinions
The main advantages of using expert opinions to construct operational profiles are the ready availability of these mostly internal experts and the much lower cost.
Other information sources include many of the existing product and system documentations, and product specifications and relevant standards.
Participants of Operational Profiles The operational profile should be developed
by a combination of system engineers, high-level designers, and test planners, with strong participation from product planning and marketing functions.
The involvement of customers is implicitly assumed here as a primary source of information.
Operational Profile Development Procedure: Musa-1 One OP is developed for each homogeneous
group of users or operations by following five basic steps:
Find the customer profile by defining a complete set of customer categories weighted per a usage factor.
Establish the user profile by defining user types weighted in relation to the customer groups and relative usage rates.
Operational Profile Development Procedure: Musa-1 Define the system modes and related profile
by identifying the sets of operations commonly used together and obtaining their related weights.
Determine the functional profile by identifying individual (high-level) functions and obtaining their related weights based on analysis of the system modes and environment variables.
Operational Profile Development Procedure: Musa-1 Determine the operational profile by refining
high-level functions into detailed operations and obtaining their usage probabilities.
An Example: Customer Profile
Customer Type Weight
corporation 0.5
government 0.4
education 0.05
other 0.05
An Example: User Profile
User User Profile by Customer Type Overall
Type ctype com gov edu etc User
weight 0.5 0.4 0.05 0.05 Profile
end user 0.8 0.9 0.9 0.7 0.84
dba 0.02 0.02 0.02 0.02 0.02
programmer 0.18 _ _ 0.28 0.104
third party _ 0.08 0.08 _ 0.036
Operational Profile Development Procedure: Musa-2 Musa-1 is suitable for large software systems. Musa-2 is suitable for smaller products or
ones with more homogeneous user population.
One operational profile is constructed for each operational mode, similar to the system mode in Musa-1.
There are 5 generic steps.
Operational Profile Development Procedure: Musa-2 Identify initiators of operations. Determine tabular or graphical
representation. Create operations lists. Determine the occurrence rate. Determine the occurrence probability.
Identify Initiators of Operations These initiators include human users as well
as other software, hardware, network, etc. Customer types and user types. External systems and system under study.
Create Operations Lists
Generate an operations list for each initiator type.
These lists can consolidated into a comprehensive list.
Determine the Occurrence Rate Actual measurement is typically used to
obtain data about the occurrence rate or frequency for individual operations in the above consolidated operations list and tabulated.
Determine the Occurrence Probability The occurrence rates are normalized to
obtain the occurrence probabilities. The occurrence probabilities satisfy 0 pi
1 and i pi = 1.
An Example
Web usage by a group of users. Stage 1: Starting the browser with two
choices: either on the default starting page or a customized starting page.
Stage 2: All subsequent usage is modeled as one stage, with three choices: following links in visited pages, following bookmarked pages, or a mixture of the two.
An Example
200/1000=0.2default
customized800/1000=0.8
160/200=0.8linked
bookmarked30/200=0.15
mixture10/200=0.05
defaultbookmarked
X
Y
80/800=0.1linked
bookmarked160/800=0.2
mixture560/800=0.7
defaultlinked
defaultmixture
customizedbookmarked
customizedlinked
customizedmixture
0.16
0.03
0.01
0.08
0.16
0.56
Stage 1:Start up
Stage 2:Other use
Operationsequence
Probability
Markov Chains
A Markov chain is a finite state machine with each transition associated with a probability and the set of probabilities satisfies the following two properties:
The probability pij for the transition from state i to state j only depends on the state i, i.e., is history independent.
0 pij 1 and j pij = 1.
An Example
Power-upMobileStationInitialization
MobileStationIdle
SystemAccess
Mobile StationControl onTraffic Channel
AB
D
C
p(A,B) = 1p(B,C) = 1
E
p(C,D) = 0.99 p(D,C) = 0.3
p(C,B) = 0.01
p(E,B) = 1
p(D,E) = 0.7
Unified Markov Models
A unified Markov model is a set of hierarchical Markov chains.
A state in a unified Markov model may represent a Markov chain.
An Example
TrafficChannelInitialization
WaitingforOrder
Waiting forMobile StationAnswer
ConversationRelease
E1 E2
E4
E3
p(1,2) = 0.5 p(2,3) = 0.9
E5
p(3,4) = 0.95p(4,3) = 0.5
p(1,4) = 0.5
p(2,5) = 0.1
p(4,5) = 0.5
p(3,5) =
0.05
p(5,3) = 0.2
Arc T_o
Arc T_i
Thresholds
Overall probability threshold: for complete end-to-end operations to ensure that commonly used complete operation sequences are covered and adequately tested.
State probability threshold: to ensure that frequently visited states are covered and adequately tested.
Transition probability threshold: to ensure that commonly used operation pairs are covered and adequately tested.
An Example
Power-upMobileStationInitialization
MobileStationIdle
SystemAccess
Mobile StationControl onTraffic Channel
AB
D
C
p(A,B) = 1p(B,C) = 1
E
p(C,D) = 0.99 p(D,C) = 0.3
p(C,B) = 0.01
p(E,B) = 1
p(D,E) = 0.7
ABCDE 1 1 0.99 0.7 = 0.693
ABCDEBCDC 1 1 0.99 0.7 1 1 0.99 0.3 = 0.205821