By
1 July 2009
Why Test Designing
The Quality of Testing is as good as its Test Design
Usage of formal and customized Test Specification techniques for
deriving Test Cases from the Input Documents (Test Basis) will
help in achieving the following:
Proper Coverage/Depth in testing each of the functions (/system)
Test Specifications prepared by various members in the test team will be uniform
Test Cases will be more Manageable
1 July 2009
Understanding test design techniques
Classic distinction – black-box and white box techniques
Black box - Black-box techniques (also called specification-based
techniques) are a way to derive and select test conditions or test cases
based on an analysis of the test basis documentation, whether
functional or non-functional, for a component or system without
reference to its internal structure.
White box - White-box techniques (also called structural or structure-
based techniques) are based on an analysis of the internal structure of
the component or system.
Black box testing may be further classified into specification based or
experience based techniques.
1 July 2009
Categories of test design techniques
Requirement based techniques:
– Models, either formal or informal, are used for the specification of
the problem to be solved, the software or its components.
– From these models test cases can be derived systematically.
Structure based techniques:
– Information about how the software is constructed is used to derive
the test cases, for example, code and design.
– The extent of coverage of the software can be measured for
existing test cases, and further test cases can be derived
systematically to increase coverage.
Experience based techniques:
– The knowledge and experience of people are used to derive the
test cases.
knowledge of testers, developers, users and other
stakeholders about the software, its usage and its
environment;
knowledge about likely defects and their distribution.
1 July 2009
Requirement based or black box techniques
Equivalence partitioning:
Inputs to the software or system are divided into groups that are
expected to exhibit similar behavior, so they are likely to be
processed in the same way. Equivalence partitions (or classes) can
be found for both valid data and invalid data, i.e. values that should
be rejected. Partitions can also be identified for outputs, internal
values, time-related values (e.g. before or after an event) and for
interface parameters (e.g. during integration testing). Tests can be
designed to cover partitions. Equivalence partitioning (EP) is
applicable at all levels of testing.
Equivalence partitioning as a technique can be used to achieve
input and output coverage. It can be applied to human input, input
via interfaces to a system, or interface parameters in integration
testing.
1 July 2009
Requirement based or black box techniques
Boundary value analysis:
Behavior at the edge of each equivalence partition is more likely to
be incorrect, so boundaries are an area where testing is likely to
yield defects. The maximum and minimum values of a partition are
its boundary values. A boundary value for a valid partition is a valid
boundary value; the boundary of an invalid partition is an invalid
boundary value. Tests can be designed to cover both valid and
invalid boundary values. When designing test cases, a value on
each boundary is chosen.
Boundary value analysis can be applied at all test levels. It is
relatively easy to apply and its defect finding capability is high;
detailed specifications are helpful. This technique is often
considered an extension of equivalence partitioning and can be
used on input by humans as well as, for example, on timing or
table boundaries. Boundary values may also be used for test data
selection.
1 July 2009
Requirement based or black box techniques
Decision table testing:
Decision tables are a good way to capture system requirements
that contain logical conditions, and to document internal system
design. They may be used to record complex business rules that a
system is to implement. The specification is analyzed, and
conditions and actions of the system are identified.
The input conditions and actions are most often stated in such a
way that they can either be true or false (Boolean). The decision
table contains the triggering conditions, often combinations of true
and false for all input conditions, and the resulting actions for each
combination of conditions. Each column of the table corresponds to
a business rule that defines a unique combination of conditions that
result in the execution of the actions associated with that rule. The
coverage standard commonly used with decision table testing is to
have at least one test per column, which typically involves covering
all combinations of triggering conditions.
The strength of decision table testing is that it creates
combinations of conditions that might not otherwise have been
exercised during testing. It may be applied to all situations when
the action of the software depends on several logical decisions.
1 July 2009
Requirement based or black box techniques
State- transition testing:
A system may exhibit a different response depending on current
conditions or previous history (its state). In this case, that aspect of
the system can be shown as a state transition diagram. It allows
the tester to view the software in terms of its states, transitions
between states, the inputs or events that trigger state changes
(transitions) and the actions which may result from those
transitions. The states of the system or object under test are
separate, identifiable and finite in number. A state table shows the
relationship between the states and inputs, and can highlight
possible transitions that are invalid. Tests can be designed to cover
a typical sequence of states, to cover every state, to exercise every
transition, to exercise specific sequences of transitions or to test
invalid transitions.
State transition testing is much used within the embedded software
industry and technical automation in general. However, the
technique is also suitable for modeling a business object having
specific states or testing screen-dialogue flows (e.g. for internet
applications or business scenarios).
1 July 2009
Requirement based or black box techniques
Use-case testing:
Tests can be specified from use cases or business scenarios. A
use case describes interactions between actors, including users
and the system, which produce a result of value to a system user.
Each use case has preconditions, which need to be met for a use
case to work successfully. Each use case terminates with post-
conditions, which are the observable results and final state of the
system after the use case has been completed. A use case usually
has a mainstream (i.e. most likely) scenario, and sometimes
alternative branches.
Use cases describe the “process flows” through a system based on
its actual likely use, so the test cases derived from use cases are
most useful in uncovering defects in the process flows during real-
world use of the system. Use cases, often referred to as scenarios,
are very useful for designing acceptance tests with customer/user
participation. They also help uncover integration defects caused by
the interaction and interference of different components, which
individual component testing would not see.
1 July 2009
Structure based or white box techniques
Structure-based testing/white-box testing is based on an identified
structure of the software or system, as seen in the following examples:
Component level: the structure is that of the code itself, i.e.
statements, decisions or branches.
Integration level: the structure may be a call tree (a diagram in
which modules call other modules).
System level: the structure may be a menu structure, business
process or web page structure.
1 July 2009
Structure based or white box techniques
Statement testing and coverage
In component testing, statement coverage is the assessment of the
percentage of executable statements that have been exercised by a test
case suite. Statement testing derives test cases to execute specific
statements, normally to increase statement coverage.
Decision testing and coverage
Decision coverage, related to branch testing, is the assessment of the
percentage of decision outcomes (e.g. the True and False options of an IF
statement) that have been exercised by a test case suite. Decision testing
derives test cases to execute specific decision outcomes, normally to
increase decision coverage.
Decision testing is a form of control flow testing as it generates a specific
flow of control through the decision points. Decision coverage is stronger
than statement coverage: 100% decision coverage guarantees 100%
statement coverage, but not vice versa.
Other structure based techniques
There are stronger levels of structural coverage beyond decision coverage,
for example, condition coverage and multiple condition coverage. The
concept of coverage can also be applied at other test levels (e.g. at
integration level) where the percentage of modules, components or classes
that have been exercised by a test case suite could be expressed as module,
component or class coverage. Tool support is useful for the structural testing
of code.
1 July 2009
Illustration of Black-box techniques
•Equivalent Partitioning
•Boundary Value Analysis
•Cause and Effect graphing
•Decision Table based Testing
1 July 2009
Guidelines for equivalence classes
1. If an input condition specifies range,
• one valid and two invalid equivalence classes are needed
2. If a condition requires a specific value,
• then one valid and two invalid equivalence classes are needed
3. If an input condition specifies a member of a set,
• one valid and one invalid equivalence class are needed
4. If an input condition is Boolean,
• one valid and one invalid class are needed
1 July 2009
Illustration for EP
EP may be best explained with an example of a function which has
the pass parameter "month" of a date. The valid range for the
month is 1 to 12, standing for January to December. This valid
range is called a partition. In this example there are two further
partitions of invalid ranges.
The first invalid partition would be <= 0 and the second invalid
partition would be >= 13.
.... -2 -1 0 1 ...................12 13 14 15 .....
--------------------|-------------------|---------------------
Invalid partition I Valid partition Invalid partition II
Equivalence partitioning is no stand alone method to determine test
cases. It has to be supplemented by another method.
1 July 2009
Guidelines for BVA
Focus on the boundaries of the input
If input condition specifies a range bounded by a certain values, say, a and b, then test cases should include
• The values for a and b
• The values just above and just below a and b
If an input condition specifies any number of values, test cases should be
the minimum and maximum numbers,
the values just above and just below the minimum and maximum
values
1 July 2009
Illustration for BVA
If the same example of a function which has the pass parameter
"month" of a date
Valid Class is 1<= month <=12
Invalid Class 1 is month <1
Invalid Class 2 is month >12
When compared to EP, which says select any test case within a
range and any on either side of it , in BVA the emphasis is on the
„edges‟
1,2 and 11,12 for the „edges‟ of the Valid class
7( a middle value ) for the Valid Class
0 and 13 for the Invalid class
Experience has shown that when a test case on or just to one side
of a boundary of an equivalence class is selected, the probability of
detecting a fault increases.
1 July 2009
Cause-Effect Graphing
This method
•Captures the relationships between specific combinations of inputs (causes) and outputs (effects)
– Deals with specific cases,
– Avoids combinatorial explosion
– Explore combinations of possible inputs
•Causes/effects are represented as nodes of a cause effect graph
•The graph also includes a number of intermediate nodes linking causes and effects
1 July 2009
Guidelines to create CEG
1. Study the functional requirements and divide the requirements into
workable pieces
2. Identify causes and effects in the specifications
i. Causes: distinct input conditions
ii. Effects: an output condition or a system transformations.
3. Assign unique number to each cause and effect
4. Use the semantic content of the spec and transform it into a Boolean
graph
5. Annotate the graph with constrains describing combinations of
causes and/or effects
6. Convert the graph into a limited-entry decision table
7. Decision table rules are converted to test cases
1 July 2009
Illustration for CEG
Consider the following set of requirements as an
example: Requirements for Calculating Car Insurance Premiums:
• R00101 For females less than 65 years of age, the premium is $500
• R00102 For males less than 25 years of age, the premium is $3000
• R00103 For males between 25 and 64 years of age, the premium is $1000
• R00104 For anyone 65 years of age or more, the premium is $1500
Causes (input conditions)
1. Sex is Male
2. Sex is Female
3. Age is <25
4. Age is >=25 and < 65
5. Age is >= 65
Effects (output conditions)
1. Premium is $1000
2. Premium is $3000
3. Premium is $1500
4. Premium is $500
1 July 2009
Drawing the CE Graph
1 July 2009
Constraint symbols in CEG
1 July 2009
Example of “O” constraint
CEG #3 for the
Calculating Car Insurance
Premiums example
would be annotated with
a “one and only one”
constraint between
causes 1 and 2 because
the Sex cause has to be
either “Male” or “Female”
but it can never be both.
1 July 2009
Drawing the Decision Table
A decision table is consists of a number of columns (rules) that comprise all
test situations.
1 July 2009
Deriving the Test Cases
Some CEGs may result in more than one test case being created. For
example, CEG #3 resulted in columns 3 and 4 and CEG#4 resulted in
columns 5 and 6 in the decision table. The columns in the decision table are
converted into test cases.
1 July 2009
Guidelines for Decision Table Technique
The below rules can be followed for Test Case Generation
• Conditions: inputs
• Actions: outputs
• Rules: test cases
The above conditions have the following
implications
– Rules are complete (i.e., every combination of decision table values including default combinations are inherent in the decision table)
– The rules are consistent (i.e, there is not two actions for the same combinations of conditions)
1 July 2009
Orthogonal Array Testing Technique
Orthogonal Array Testing is a statistical technique implemented by
Taguchi. It is a statistical way of testing pair-wise interactions. It
provides a uniformly distributed representative coverage of all
variable pair combinations.
It is useful in Integration testing of software components in complex
applications like E-Com.
Sufficient knowledge about the system under test combined with
the power of statistical technique, helps in providing maximum
coverage with minimum number of test cases.
1 July 2009
What is an Orthogonal Array?
Orthogonal Array is a matrix of “n” rows and “k” columns with every element being
one of the “q” symbols 0 to q-1.
The array has strength “t” if in every n*t sub matrix, the qt possible distinct rows all
appear the same number of times.
OAT is an array of values in which each column represents a variable-factor that
can take a certain set of values called levels.
Each row in an orthogonal array represents a test case.
In OAT the factors are combined pair-wise rather than representing all possible
combinations of factors and levels.
Refer to- “http://support.sas.com” for pre defined Orthogonal arrays for various values for n,k,q,t.
1 July 2009
Illustration of OAT
0 0 0
0 1 1
0 1 0
0 0 1
1 0 0
1 0 1
1 1 0
1 1 1
0 0
0 1
0 1
0 0
1 0
1 0
1 1
1 1
0 0
0 1
0 0
0 1
1 0
1 1
1 0
1 1
0 0
1 1
1 0
0 1
0 0
0 1
1 0
1 1
In every 8*2 sub matrix, the 22 possible distinct rows all appear the same number of
times. Hence by OAT, we can arrive at a minimum of 4 possible test cases which is
said to provide maximum coverage.
1 & 2 2 & 3 1 & 3
1 July 2009
Orthogonal Array- Example
Size Base Topping
Small Thin Capsicum
Medium Thick Onion
Large Pan Mushroom
Let us consider a Pizza shop offering the following varieties of Pizza
As a part of introductory offer, Pizza shop is planning to offer free Pizzas.
But the question here is “What Pizza combinations” need to be offered free?
What is the optimal number of combinations that needs to be considered to
cover all varieties?
1 July 2009
Orthogonal Array- Example
Orthogonal Array (n,k,q,t) Orthogonal Array (9,3,3,2)
Size Base Topping
Small (0) Thin (0) Capsicum (0)
Small (0) Thick (1) Onion (1)
Small (0) Pan (2) Mushroom (2)
Medium (1) Thin (0) Onion (1)
Medium (1) Thick (1) Mushroom (2)
Medium (1) Pan (2) Capsicum (0)
Large (2) Thin (0) Mushroom (2)
Large (2) Thick (1) Capsicum (0)
Large (2) Pan (2) Onion (1)
The optimal number of combinations possible are:
1 July 2009
Orthogonal Array- Practical Considerations
If in case an orthogonal array has 9 rows, 3 variables. 2 variables have 3 distinct values but the
other variable has only 2 values. In such case, the orthogonal array can be represented as :
L 9 2 3 1 2
In such cases, the combinations can be formed by selecting “any” value.
Size Base Topping
Small (0) Thin (0) Capsicum (0)
Small (0) Thick (1) Onion (1)
Small (0) Any Mushroom (2)
Medium (1) Thin (0) Onion (1)
Medium (1) Thick (1) Mushroom (2)
Medium (1) Any Capsicum (0)
Large (2) Thin (0) Mushroom (2)
Large (2) Thick (1) Capsicum (0)
Large (2) Any Onion (1)
Care needs to be taken to ensure that the combinations are practically feasible.
1 July 2009
OATS-A Statistical Optimization Technique
• How OATS help overcome the challenges of Software Testing
optimization?
– Helps optimize testing process to face the fast paced industry
with aggressive time-to-market
– Helps to reduce Test Cycle Time
– Finds maximum defects with minimal test cases
– Provides maximum coverage with minimum test cases
- 32
-
1 July 2009
POC on A Leading Insurance Company in UK
• Application: Administrator (Admin System for Aviva’s Staff Pension scheme)
• Project: Goya
• A quick case study has been done for a small part of Aviva’s Staff Pension scheme
Administrator Application that contains three independent variables ( i.e. A -Configure the
scheme , B - Issuing the policy , C - Investment calculation ) and each variable can take on
three values
– (A0- Scheme Selection, A1- Start & End date , A2- Contribution to Fund).
– (B0- Policy holder’s details, B1- Contribution, B2- Acceptance of proposal).
– (C0- Investment Amount, C1- Number of units, C2-Term of retirement ).
- 33
-
MAPPING THE FACTORS IN OA
Configure the scheme (A) Issuing the policy – (B) Investment calculation – (C) Factor 4
TC-1 Scheme selection Policy holder’s details Investment Amount
TC- 2 Scheme selection Contribution Number of units
TC- 3 Scheme selection Acceptance of proposal Term of retirement
TC- 4 Start & End date Policy holder’s details Number of units
TC- 5 Start & End date Contribution Term of retirement
TC- 6 Start & End date Acceptance of proposal Investment Amount
TC- 7 Contribution to fund Policy holder’s details Term of retirement
TC- 8 Contribution to fund Contribution Investment Amount
TC- 9 Contribution to fund Acceptance of proposal Number of units
To test all of the possible combinations, it would take a test set
containing 27 test cases (3 x 3 x 3 = 27) in Exhaustive Testing. In real
scenario, the Test analyst has written 26 test cases to test the above
piece of requirement. For designing each test case, the Test Analyst
had to spend 20-30 minutes.
1 July 2009
POC on AVIVA – Project GOYA CONTD...
- 34
-
-RESULTS
-Using the OATS technique we can reduce the number of test cases to 9 in order
to test all the pair-wise combinations.
Three suspicious combinations for three variables that are not generated by the
OATS method are included additionally. Total no of scripts are (9+3) = 12
- TEST CASE REDUCTION
The number of test cases were reduced to 12 as against 26; Which is 46% of the
total test cases.
- EFFORT REDUCTION
The Test Case generation time was also reduced by at least 4.6 hrs. Which is also
46% of the total time,
- CYCLE TIME REDUCTION
Due to decreased number of test cases, cycle time also reduced by 46%
- INCREASE PRODUCTIVITY
The average time which require for test case preparation and execution are also
reduced which in turn increases the Productivity.
1 July 2009
Testing Challenges and OATS
Too many Test Cases
Insufficient
Coverage
Testing Challenges
Too much Effort
How OATS helps!!
Reduces Effort significantly
Optimal number of Test
Cases
Optimal Coverage
Too much Time and Cost Reduces TC Time and Cost
Overall 40% to 50% Effort,
Time and Cost Reduction
with Optimal Coverage
1 July 2009
Summary-Test Case Design Techniques
Test Case designing is an important aspect for Test Coverage.
Depending on the type of testing done ,
Requirement-based Test design techniques
Code-based Test design techniques
For Functionality / Black Box Testing, Requirements are the only source
for Test Specifications.
» Equivalence Class Partitioning
» Boundary Value Analysis
» Cause-Effect Graphing
» Decision Table Technique
» Orthogonal Array Technique
1 July 2009
Exercise-1
Prepare blackbox test cases on how you would test the Yahoo
webpage
Prepare whitebox test cases for this simple C program#include <stdio.h>
#define LAST 10
int main()
{ int i, sum = 0;
for ( i = 1; i <= LAST; i++ ) {
sum += i;
} /*-for-*/
printf("sum = %d\n", sum);
return 0;
}
1 July 2009
Exercise-2
Annual Exam Mark Processing for Orlinda Pierce Memorial Hr.Sec. School
Final Grade Calculation
The Candidate has got "Final Grade" grade in the Annual Examination
Enter Student Roll No
Enter Final Exam Marks
Enter Class Work Marks
Total Marks
Final Grade
1 July 2009
Exercise 2-Contd…
Functional Specification for the “Final Grade Calculation” screen
The Final Exam mark is out of 75 and the Class Work mark is out of 25. The Total
Mark is calculated as the sum of the Final Exam Mark and the Class Work Mark.
The Final Grade is calculated in the range ‘A’ to ‘D’. The classification of Final
Grade is done as below
Total Marks greater than or equal to 70 - 'A'
Total Marks greater than or equal to 50, but less than 70 - 'B'
Total Marks greater than or equal to 30, but less than 50 - 'C'
Total Marks less than 30 - 'D'
Where a Total Mark outside its expected range is derived then a fault message
('FM') is generated.
The calculation has to be done upon entering a valid Roll No which is a 5 digit
character data with the starting range of 10000. All other inputs are passed as
integers.
Using EP, BVA and CEG derive at the Test Cases for the above Specification.
1 July 2009 TC
S
Int
Experience-Based Testing
Use of experience to design test cases
Experience can include
Domain knowledge
Knowledge of developers involved
Knowledge of typical problems
Two main types
Ad Hoc Testing
Exploratory Testing
1 July 2009 TC
S
Int
Experience-Based Testing (contd)
• Ad-hoc Testing
– Informal testing
– No preparation
– Not repeatable
– Cannot be tracked
• Exploratory Testing
– Informal
– Actually involves test design and control
– Useful when no specification is available
– Notes are taken and progress tracked
The Art of Test Case Writing
1 July 2009 43
Objective and Importance of a Test Case
The basic objective of writing test cases is to ensure complete test
coverage of the application.
The most extensive effort in preparing to test a software, is writing
test cases.
Gives better reliability in estimating the test effort
Improves productivity during test execution by reducing the
“understanding” time during execution
Writing effective test cases is a skill and that can be achieved by
experience and in-depth study of the application on which test cases
are being written.
1 July 2009 44
Importance of Test Case-Contd.
Sotware Testing has been recognised as a separate technical
disciple in the software development life cycle
A structured and process oriented approach is recommended in
every phase of the test process
Documenting the test cases prior to test execution ensures that the
tester does the „homework‟ and is prepared for the „attack‟ on the
Application Under Test
Breaking down the Test Requirements into Test Scenarios and Test
Cases would help the testers avoid missing out certain test conditions
1 July 2009 45
What is a Test Case?
It is the smallest unit of Testing
A test case is a detailed procedure that fully tests a feature or an
aspect of a feature. Whereas the test plan describes what to test, a
test case describes how to perform a particular test.
A test case has components that describes an input, action or event
and an expected response, to determine if a feature of an application
is working correctly.”
Test cases must be written by a team member who thoroughly
understands the function being tested.
1 July 2009 46
What is a Good Test Case
A test case which has the following characteristics is considered to be
a good test case
Accurate -addresses the objective
Economical-Limited precise steps
Maintainable , Reusable
Maps to the logical flow of the application
Both Forward & Backward traceable-to a requirement as well as
defect
Self Explanatory and requires no help in understanding during
execution
Self Cleaning & Brief comments in case of Automation test
scripts
1 July 2009 47
Elements of a Test Case
Every test case must have the following details
Anatomy of a Test Case
Test Case ID
Requirement # / Section:
Objective: What is to be verified?
Assumptions & Prerequisites
Steps to be executed:
Test data(if any): Variables and their values
Expected result:
Status: Pass or Fail with details on Defect ID and proofs
[o/p files, printouts, screenshots(optional)]
Comments:
Any CMMi company would have defined templates and standards
to be adhered to while writing test cases.
1 July 2009
Language usage in Test Case writing
48
Use Simple and Easy-to-Understand language
Use Active voice while writing test cases For eg.Click on OK button
Enter the data in screen1
Choose the option1
Navigate to the account Summary page.
Use words like “Verify” for starting any sentence in Test Case
description For eg.Verify whether the account information gets displayed on clicking the “Account
Summary” menu
Use words like “Must” and use Future Tense for Expected ResultsThe application must display the account information screen
An error message must be displayed on entering special characters
1 July 2009 49
Some Tips on Test Case Writing
Don't write long test cases(limit to maximum of 10 steps)
Avoid incoherence. Follow a logical flow from one test case to
another.
Don't skip steps assuming that it is “understood”
Don't confuse between Action & Results For eg.“The button is pressed”
Don't write test cases as paragraphs in a word document. Use
Table/Matrix format or any Spreadsheet format.
Follow a logical flow , but start writing test cases for those
requirements that are frozen and make a mark like TBD for those
unclear sections.
1 July 2009 50
Sample Condition to write a test case
I. First and last name fields are mandatory and are limited to 50 characters
each. The fields will not accept numerical values and the following
characters: `~!@#$%^&*()_:";'{}[]+<>?,./
II. Respective Error messages are thrown on NOT meeting the above conditions
III. User will be registered successfully on entering the correct data
User Registration
Enter the First and Last Name and Click the Submit Button
First Name None
Submit
Second Name None
Thank You
Recommended