By

1 July 2009

Why Test Designing

The Quality of Testing is as good as its Test Design

Usage of formal and customized Test Specification techniques for

deriving Test Cases from the Input Documents (Test Basis) will

help in achieving the following:

Proper Coverage/Depth in testing each of the functions (/system)

Test Specifications prepared by various members in the test team will be uniform

Test Cases will be more Manageable

1 July 2009

Understanding test design techniques

Classic distinction – black-box and white box techniques

Black box - Black-box techniques (also called specification-based

techniques) are a way to derive and select test conditions or test cases

based on an analysis of the test basis documentation, whether

functional or non-functional, for a component or system without

reference to its internal structure.

White box - White-box techniques (also called structural or structure-

based techniques) are based on an analysis of the internal structure of

the component or system.

Black box testing may be further classified into specification based or

experience based techniques.

1 July 2009

Categories of test design techniques

Requirement based techniques:

– Models, either formal or informal, are used for the specification of

the problem to be solved, the software or its components.

– From these models test cases can be derived systematically.

Structure based techniques:

– Information about how the software is constructed is used to derive

the test cases, for example, code and design.

– The extent of coverage of the software can be measured for

existing test cases, and further test cases can be derived

systematically to increase coverage.

Experience based techniques:

– The knowledge and experience of people are used to derive the

test cases.

knowledge of testers, developers, users and other

stakeholders about the software, its usage and its

environment;

knowledge about likely defects and their distribution.

1 July 2009

Requirement based or black box techniques

Equivalence partitioning:

Inputs to the software or system are divided into groups that are

expected to exhibit similar behavior, so they are likely to be

processed in the same way. Equivalence partitions (or classes) can

be found for both valid data and invalid data, i.e. values that should

be rejected. Partitions can also be identified for outputs, internal

values, time-related values (e.g. before or after an event) and for

interface parameters (e.g. during integration testing). Tests can be

designed to cover partitions. Equivalence partitioning (EP) is

applicable at all levels of testing.

Equivalence partitioning as a technique can be used to achieve

input and output coverage. It can be applied to human input, input

via interfaces to a system, or interface parameters in integration

testing.

1 July 2009

Requirement based or black box techniques

Boundary value analysis:

Behavior at the edge of each equivalence partition is more likely to

be incorrect, so boundaries are an area where testing is likely to

yield defects. The maximum and minimum values of a partition are

its boundary values. A boundary value for a valid partition is a valid

boundary value; the boundary of an invalid partition is an invalid

boundary value. Tests can be designed to cover both valid and

invalid boundary values. When designing test cases, a value on

each boundary is chosen.

Boundary value analysis can be applied at all test levels. It is

relatively easy to apply and its defect finding capability is high;

detailed specifications are helpful. This technique is often

considered an extension of equivalence partitioning and can be

used on input by humans as well as, for example, on timing or

table boundaries. Boundary values may also be used for test data

selection.

1 July 2009

Requirement based or black box techniques

Decision table testing:

Decision tables are a good way to capture system requirements

that contain logical conditions, and to document internal system

design. They may be used to record complex business rules that a

system is to implement. The specification is analyzed, and

conditions and actions of the system are identified.

The input conditions and actions are most often stated in such a

way that they can either be true or false (Boolean). The decision

table contains the triggering conditions, often combinations of true

and false for all input conditions, and the resulting actions for each

combination of conditions. Each column of the table corresponds to

a business rule that defines a unique combination of conditions that

result in the execution of the actions associated with that rule. The

coverage standard commonly used with decision table testing is to

have at least one test per column, which typically involves covering

all combinations of triggering conditions.

The strength of decision table testing is that it creates

combinations of conditions that might not otherwise have been

exercised during testing. It may be applied to all situations when

the action of the software depends on several logical decisions.

1 July 2009

Requirement based or black box techniques

State- transition testing:

A system may exhibit a different response depending on current

conditions or previous history (its state). In this case, that aspect of

the system can be shown as a state transition diagram. It allows

the tester to view the software in terms of its states, transitions

between states, the inputs or events that trigger state changes

(transitions) and the actions which may result from those

transitions. The states of the system or object under test are

separate, identifiable and finite in number. A state table shows the

relationship between the states and inputs, and can highlight

possible transitions that are invalid. Tests can be designed to cover

a typical sequence of states, to cover every state, to exercise every

transition, to exercise specific sequences of transitions or to test

invalid transitions.

State transition testing is much used within the embedded software

industry and technical automation in general. However, the

technique is also suitable for modeling a business object having

specific states or testing screen-dialogue flows (e.g. for internet

applications or business scenarios).

1 July 2009

Requirement based or black box techniques

Use-case testing:

Tests can be specified from use cases or business scenarios. A

use case describes interactions between actors, including users

and the system, which produce a result of value to a system user.

Each use case has preconditions, which need to be met for a use

case to work successfully. Each use case terminates with post-

conditions, which are the observable results and final state of the

system after the use case has been completed. A use case usually

has a mainstream (i.e. most likely) scenario, and sometimes

alternative branches.

Use cases describe the “process flows” through a system based on

its actual likely use, so the test cases derived from use cases are

most useful in uncovering defects in the process flows during real-

world use of the system. Use cases, often referred to as scenarios,

are very useful for designing acceptance tests with customer/user

participation. They also help uncover integration defects caused by

the interaction and interference of different components, which

individual component testing would not see.

1 July 2009

Structure based or white box techniques

Structure-based testing/white-box testing is based on an identified

structure of the software or system, as seen in the following examples:

Component level: the structure is that of the code itself, i.e.

statements, decisions or branches.

Integration level: the structure may be a call tree (a diagram in

which modules call other modules).

System level: the structure may be a menu structure, business

process or web page structure.

1 July 2009

Structure based or white box techniques

Statement testing and coverage

In component testing, statement coverage is the assessment of the

percentage of executable statements that have been exercised by a test

case suite. Statement testing derives test cases to execute specific

statements, normally to increase statement coverage.

Decision testing and coverage

Decision coverage, related to branch testing, is the assessment of the

percentage of decision outcomes (e.g. the True and False options of an IF

statement) that have been exercised by a test case suite. Decision testing

derives test cases to execute specific decision outcomes, normally to

increase decision coverage.

Decision testing is a form of control flow testing as it generates a specific

flow of control through the decision points. Decision coverage is stronger

than statement coverage: 100% decision coverage guarantees 100%

statement coverage, but not vice versa.

Other structure based techniques

There are stronger levels of structural coverage beyond decision coverage,

for example, condition coverage and multiple condition coverage. The

concept of coverage can also be applied at other test levels (e.g. at

integration level) where the percentage of modules, components or classes

that have been exercised by a test case suite could be expressed as module,

component or class coverage. Tool support is useful for the structural testing

of code.

1 July 2009

Illustration of Black-box techniques

•Equivalent Partitioning

•Boundary Value Analysis

•Cause and Effect graphing

•Decision Table based Testing

1 July 2009

Guidelines for equivalence classes

1. If an input condition specifies range,

• one valid and two invalid equivalence classes are needed

2. If a condition requires a specific value,

• then one valid and two invalid equivalence classes are needed

3. If an input condition specifies a member of a set,

• one valid and one invalid equivalence class are needed

4. If an input condition is Boolean,

• one valid and one invalid class are needed

1 July 2009

Illustration for EP

EP may be best explained with an example of a function which has

the pass parameter "month" of a date. The valid range for the

month is 1 to 12, standing for January to December. This valid

range is called a partition. In this example there are two further

partitions of invalid ranges.

The first invalid partition would be <= 0 and the second invalid

partition would be >= 13.

.... -2 -1 0 1 ...................12 13 14 15 .....

--------------------|-------------------|---------------------

Invalid partition I Valid partition Invalid partition II

Equivalence partitioning is no stand alone method to determine test

cases. It has to be supplemented by another method.

1 July 2009

Guidelines for BVA

Focus on the boundaries of the input

If input condition specifies a range bounded by a certain values, say, a and b, then test cases should include

• The values for a and b

• The values just above and just below a and b

If an input condition specifies any number of values, test cases should be

the minimum and maximum numbers,

the values just above and just below the minimum and maximum

values

1 July 2009

Illustration for BVA

If the same example of a function which has the pass parameter

"month" of a date

Valid Class is 1<= month <=12

Invalid Class 1 is month <1

Invalid Class 2 is month >12

When compared to EP, which says select any test case within a

range and any on either side of it , in BVA the emphasis is on the

„edges‟

1,2 and 11,12 for the „edges‟ of the Valid class

7( a middle value ) for the Valid Class

0 and 13 for the Invalid class

Experience has shown that when a test case on or just to one side

of a boundary of an equivalence class is selected, the probability of

detecting a fault increases.

1 July 2009

Cause-Effect Graphing

This method

•Captures the relationships between specific combinations of inputs (causes) and outputs (effects)

– Deals with specific cases,

– Avoids combinatorial explosion

– Explore combinations of possible inputs

•Causes/effects are represented as nodes of a cause effect graph

•The graph also includes a number of intermediate nodes linking causes and effects

1 July 2009

Guidelines to create CEG

1. Study the functional requirements and divide the requirements into

workable pieces

2. Identify causes and effects in the specifications

i. Causes: distinct input conditions

ii. Effects: an output condition or a system transformations.

3. Assign unique number to each cause and effect

4. Use the semantic content of the spec and transform it into a Boolean

graph

5. Annotate the graph with constrains describing combinations of

causes and/or effects

6. Convert the graph into a limited-entry decision table

7. Decision table rules are converted to test cases

1 July 2009

Illustration for CEG

Consider the following set of requirements as an

example: Requirements for Calculating Car Insurance Premiums:

• R00101 For females less than 65 years of age, the premium is $500

• R00102 For males less than 25 years of age, the premium is $3000

• R00103 For males between 25 and 64 years of age, the premium is $1000

• R00104 For anyone 65 years of age or more, the premium is $1500

Causes (input conditions)

1. Sex is Male

2. Sex is Female

3. Age is <25

4. Age is >=25 and < 65

5. Age is >= 65

Effects (output conditions)

1. Premium is $1000

2. Premium is $3000

3. Premium is $1500

4. Premium is $500

1 July 2009

Drawing the CE Graph

1 July 2009

Constraint symbols in CEG

1 July 2009

Example of “O” constraint

CEG #3 for the

Calculating Car Insurance

Premiums example

would be annotated with

a “one and only one”

constraint between

causes 1 and 2 because

the Sex cause has to be

either “Male” or “Female”

but it can never be both.

1 July 2009

Drawing the Decision Table

A decision table is consists of a number of columns (rules) that comprise all

test situations.

1 July 2009

Deriving the Test Cases

Some CEGs may result in more than one test case being created. For

example, CEG #3 resulted in columns 3 and 4 and CEG#4 resulted in

columns 5 and 6 in the decision table. The columns in the decision table are

converted into test cases.

1 July 2009

Guidelines for Decision Table Technique

The below rules can be followed for Test Case Generation

• Conditions: inputs

• Actions: outputs

• Rules: test cases

The above conditions have the following

implications

– Rules are complete (i.e., every combination of decision table values including default combinations are inherent in the decision table)

– The rules are consistent (i.e, there is not two actions for the same combinations of conditions)

1 July 2009

Orthogonal Array Testing Technique

Orthogonal Array Testing is a statistical technique implemented by

Taguchi. It is a statistical way of testing pair-wise interactions. It

provides a uniformly distributed representative coverage of all

variable pair combinations.

It is useful in Integration testing of software components in complex

applications like E-Com.

Sufficient knowledge about the system under test combined with

the power of statistical technique, helps in providing maximum

coverage with minimum number of test cases.

1 July 2009

What is an Orthogonal Array?

Orthogonal Array is a matrix of “n” rows and “k” columns with every element being

one of the “q” symbols 0 to q-1.

The array has strength “t” if in every n*t sub matrix, the qt possible distinct rows all

appear the same number of times.

OAT is an array of values in which each column represents a variable-factor that

can take a certain set of values called levels.

Each row in an orthogonal array represents a test case.

In OAT the factors are combined pair-wise rather than representing all possible

combinations of factors and levels.

Refer to- “http://support.sas.com” for pre defined Orthogonal arrays for various values for n,k,q,t.

1 July 2009

Illustration of OAT

0 0 0

0 1 1

0 1 0

0 0 1

1 0 0

1 0 1

1 1 0

1 1 1

0 0

0 1

0 1

0 0

1 0

1 0

1 1

1 1

0 0

0 1

0 0

0 1

1 0

1 1

1 0

1 1

0 0

1 1

1 0

0 1

0 0

0 1

1 0

1 1

In every 8*2 sub matrix, the 22 possible distinct rows all appear the same number of

times. Hence by OAT, we can arrive at a minimum of 4 possible test cases which is

said to provide maximum coverage.

1 & 2 2 & 3 1 & 3

1 July 2009

Orthogonal Array- Example

Size Base Topping

Small Thin Capsicum

Medium Thick Onion

Large Pan Mushroom

Let us consider a Pizza shop offering the following varieties of Pizza

As a part of introductory offer, Pizza shop is planning to offer free Pizzas.

But the question here is “What Pizza combinations” need to be offered free?

What is the optimal number of combinations that needs to be considered to

cover all varieties?

1 July 2009

Orthogonal Array- Example

Orthogonal Array (n,k,q,t) Orthogonal Array (9,3,3,2)

Size Base Topping

Small (0) Thin (0) Capsicum (0)

Small (0) Thick (1) Onion (1)

Small (0) Pan (2) Mushroom (2)

Medium (1) Thin (0) Onion (1)

Medium (1) Thick (1) Mushroom (2)

Medium (1) Pan (2) Capsicum (0)

Large (2) Thin (0) Mushroom (2)

Large (2) Thick (1) Capsicum (0)

Large (2) Pan (2) Onion (1)

The optimal number of combinations possible are:

1 July 2009

Orthogonal Array- Practical Considerations

If in case an orthogonal array has 9 rows, 3 variables. 2 variables have 3 distinct values but the

other variable has only 2 values. In such case, the orthogonal array can be represented as :

L 9 2 3 1 2

In such cases, the combinations can be formed by selecting “any” value.

Size Base Topping

Small (0) Thin (0) Capsicum (0)

Small (0) Thick (1) Onion (1)

Small (0) Any Mushroom (2)

Medium (1) Thin (0) Onion (1)

Medium (1) Thick (1) Mushroom (2)

Medium (1) Any Capsicum (0)

Large (2) Thin (0) Mushroom (2)

Large (2) Thick (1) Capsicum (0)

Large (2) Any Onion (1)

Care needs to be taken to ensure that the combinations are practically feasible.

1 July 2009

OATS-A Statistical Optimization Technique

• How OATS help overcome the challenges of Software Testing

optimization?

– Helps optimize testing process to face the fast paced industry

with aggressive time-to-market

– Helps to reduce Test Cycle Time

– Finds maximum defects with minimal test cases

– Provides maximum coverage with minimum test cases

- 32

-

1 July 2009

POC on A Leading Insurance Company in UK

• Application: Administrator (Admin System for Aviva’s Staff Pension scheme)

• Project: Goya

• A quick case study has been done for a small part of Aviva’s Staff Pension scheme

Administrator Application that contains three independent variables ( i.e. A -Configure the

scheme , B - Issuing the policy , C - Investment calculation ) and each variable can take on

three values

– (A0- Scheme Selection, A1- Start & End date , A2- Contribution to Fund).

– (B0- Policy holder’s details, B1- Contribution, B2- Acceptance of proposal).

– (C0- Investment Amount, C1- Number of units, C2-Term of retirement ).

- 33

-

MAPPING THE FACTORS IN OA

Configure the scheme (A) Issuing the policy – (B) Investment calculation – (C) Factor 4

TC-1 Scheme selection Policy holder’s details Investment Amount

TC- 2 Scheme selection Contribution Number of units

TC- 3 Scheme selection Acceptance of proposal Term of retirement

TC- 4 Start & End date Policy holder’s details Number of units

TC- 5 Start & End date Contribution Term of retirement

TC- 6 Start & End date Acceptance of proposal Investment Amount

TC- 7 Contribution to fund Policy holder’s details Term of retirement

TC- 8 Contribution to fund Contribution Investment Amount

TC- 9 Contribution to fund Acceptance of proposal Number of units

To test all of the possible combinations, it would take a test set

containing 27 test cases (3 x 3 x 3 = 27) in Exhaustive Testing. In real

scenario, the Test analyst has written 26 test cases to test the above

piece of requirement. For designing each test case, the Test Analyst

had to spend 20-30 minutes.

1 July 2009

POC on AVIVA – Project GOYA CONTD...

- 34

-

-RESULTS

-Using the OATS technique we can reduce the number of test cases to 9 in order

to test all the pair-wise combinations.

Three suspicious combinations for three variables that are not generated by the

OATS method are included additionally. Total no of scripts are (9+3) = 12

- TEST CASE REDUCTION

The number of test cases were reduced to 12 as against 26; Which is 46% of the

total test cases.

- EFFORT REDUCTION

The Test Case generation time was also reduced by at least 4.6 hrs. Which is also

46% of the total time,

- CYCLE TIME REDUCTION

Due to decreased number of test cases, cycle time also reduced by 46%

- INCREASE PRODUCTIVITY

The average time which require for test case preparation and execution are also

reduced which in turn increases the Productivity.

1 July 2009

Testing Challenges and OATS

Too many Test Cases

Insufficient

Coverage

Testing Challenges

Too much Effort

How OATS helps!!

Reduces Effort significantly

Optimal number of Test

Cases

Optimal Coverage

Too much Time and Cost Reduces TC Time and Cost

Overall 40% to 50% Effort,

Time and Cost Reduction

with Optimal Coverage

1 July 2009

Summary-Test Case Design Techniques

Test Case designing is an important aspect for Test Coverage.

Depending on the type of testing done ,

Requirement-based Test design techniques

Code-based Test design techniques

For Functionality / Black Box Testing, Requirements are the only source

for Test Specifications.

» Equivalence Class Partitioning

» Boundary Value Analysis

» Cause-Effect Graphing

» Decision Table Technique

» Orthogonal Array Technique

1 July 2009

Exercise-1

Prepare blackbox test cases on how you would test the Yahoo

webpage

Prepare whitebox test cases for this simple C program#include <stdio.h>

#define LAST 10

int main()

{ int i, sum = 0;

for ( i = 1; i <= LAST; i++ ) {

sum += i;

} /*-for-*/

printf("sum = %d\n", sum);

return 0;

}

1 July 2009

Exercise-2

Annual Exam Mark Processing for Orlinda Pierce Memorial Hr.Sec. School

Final Grade Calculation

The Candidate has got "Final Grade" grade in the Annual Examination

Enter Student Roll No

Enter Final Exam Marks

Enter Class Work Marks

Total Marks

Final Grade

1 July 2009

Exercise 2-Contd…

Functional Specification for the “Final Grade Calculation” screen

The Final Exam mark is out of 75 and the Class Work mark is out of 25. The Total

Mark is calculated as the sum of the Final Exam Mark and the Class Work Mark.

The Final Grade is calculated in the range ‘A’ to ‘D’. The classification of Final

Grade is done as below

Total Marks greater than or equal to 70 - 'A'

Total Marks greater than or equal to 50, but less than 70 - 'B'

Total Marks greater than or equal to 30, but less than 50 - 'C'

Total Marks less than 30 - 'D'

Where a Total Mark outside its expected range is derived then a fault message

('FM') is generated.

The calculation has to be done upon entering a valid Roll No which is a 5 digit

character data with the starting range of 10000. All other inputs are passed as

integers.

Using EP, BVA and CEG derive at the Test Cases for the above Specification.

1 July 2009 TC

S

Int

Experience-Based Testing

Use of experience to design test cases

Experience can include

Domain knowledge

Knowledge of developers involved

Knowledge of typical problems

Two main types

Ad Hoc Testing

Exploratory Testing

1 July 2009 TC

S

Int

Experience-Based Testing (contd)

• Ad-hoc Testing

– Informal testing

– No preparation

– Not repeatable

– Cannot be tracked

• Exploratory Testing

– Informal

– Actually involves test design and control

– Useful when no specification is available

– Notes are taken and progress tracked

The Art of Test Case Writing

1 July 2009 43

Objective and Importance of a Test Case

The basic objective of writing test cases is to ensure complete test

coverage of the application.

The most extensive effort in preparing to test a software, is writing

test cases.

Gives better reliability in estimating the test effort

Improves productivity during test execution by reducing the

“understanding” time during execution

Writing effective test cases is a skill and that can be achieved by

experience and in-depth study of the application on which test cases

are being written.

1 July 2009 44

Importance of Test Case-Contd.

Sotware Testing has been recognised as a separate technical

disciple in the software development life cycle

A structured and process oriented approach is recommended in

every phase of the test process

Documenting the test cases prior to test execution ensures that the

tester does the „homework‟ and is prepared for the „attack‟ on the

Application Under Test

Breaking down the Test Requirements into Test Scenarios and Test

Cases would help the testers avoid missing out certain test conditions

1 July 2009 45

What is a Test Case?

It is the smallest unit of Testing

A test case is a detailed procedure that fully tests a feature or an

aspect of a feature. Whereas the test plan describes what to test, a

test case describes how to perform a particular test.

A test case has components that describes an input, action or event

and an expected response, to determine if a feature of an application

is working correctly.”

Test cases must be written by a team member who thoroughly

understands the function being tested.

1 July 2009 46

What is a Good Test Case

A test case which has the following characteristics is considered to be

a good test case

Accurate -addresses the objective

Economical-Limited precise steps

Maintainable , Reusable

Maps to the logical flow of the application

Both Forward & Backward traceable-to a requirement as well as

defect

Self Explanatory and requires no help in understanding during

execution

Self Cleaning & Brief comments in case of Automation test

scripts

1 July 2009 47

Elements of a Test Case

Every test case must have the following details

Anatomy of a Test Case

Test Case ID

Requirement # / Section:

Objective: What is to be verified?

Assumptions & Prerequisites

Steps to be executed:

Test data(if any): Variables and their values

Expected result:

Status: Pass or Fail with details on Defect ID and proofs

[o/p files, printouts, screenshots(optional)]

Comments:

Any CMMi company would have defined templates and standards

to be adhered to while writing test cases.

1 July 2009

Language usage in Test Case writing

48

Use Simple and Easy-to-Understand language

Use Active voice while writing test cases For eg.Click on OK button

Enter the data in screen1

Choose the option1

Navigate to the account Summary page.

Use words like “Verify” for starting any sentence in Test Case

description For eg.Verify whether the account information gets displayed on clicking the “Account

Summary” menu

Use words like “Must” and use Future Tense for Expected ResultsThe application must display the account information screen

An error message must be displayed on entering special characters

1 July 2009 49

Some Tips on Test Case Writing

Don't write long test cases(limit to maximum of 10 steps)

Avoid incoherence. Follow a logical flow from one test case to

another.

Don't skip steps assuming that it is “understood”

Don't confuse between Action & Results For eg.“The button is pressed”

Don't write test cases as paragraphs in a word document. Use

Table/Matrix format or any Spreadsheet format.

Follow a logical flow , but start writing test cases for those

requirements that are frozen and make a mark like TBD for those

unclear sections.

1 July 2009 50

Sample Condition to write a test case

I. First and last name fields are mandatory and are limited to 50 characters

each. The fields will not accept numerical values and the following

characters: `~!@#$%^&*()_:";'{}[]+<>?,./

II. Respective Error messages are thrown on NOT meeting the above conditions

III. User will be registered successfully on entering the correct data

User Registration

Enter the First and Last Name and Click the Submit Button

First Name None

Submit

Second Name None

Thank You