Software Testing Notes

Unit – 1 Notes

Important 16 Marks Questions

1. Explain the Principles of Software Testing

2. Explain error checklist for code inspection

3. Compare and contrast Blackbox and white box testing

4. Explain blackbox and white box testing in detail.

5. Explain the psychology of Software testing.

What is Software Testing?

Software testing is a process used which is to help identify the correctness, completeness and

quality of developed computer software.

An inspection or study carried out to determine the status of a client’s requirements with an aim to

identify the bugs for the application under test is referred to as software testing.

The Psychology of Testing

One of the primary causes of poor program testing is the fact that most programmers begin with a

false definition of the term. They might say:

✓ “Testing is the process of demonstrating that errors are not present.”

(Or)

✓ “The purpose of testing is to show that a program performs its intended functions

correctly.”

Thus, a more appropriate definition is this:

Testing is the process of executing a program with the intent of finding errors.

Another way of reinforcing the proper definition of testing is to analyze the use of the words

“successful” and “unsuccessful”—in particular, their use by project managers in categorizing the

results of test cases. Most project managers call a test case that did not find an error a “successful

test run,” whereas a test that discovers a new error is usually called “unsuccessful.”

A test case that finds a new error can hardly be considered unsuccessful; rather, it has proven to

be a valuable investment. An unsuccessful test case is one that causes a program to produce the

correct result without finding any errors.

For example

Consider the analogy of a person visiting a doctor because of an overall feeling of malaise. If the

doctor runs some laboratory tests that do not locate the problem, we do not call the laboratory tests

“successful”; they were unsuccessful tests in that the patient’s net worth has been reduced by the

expensive laboratory fees, the patient is still ill, and the patient may question the doctor’s ability

as a diagnostician.However, if a laboratory test determines that the patient has a peptic ulcer, the

test is successful because the doctor can now begin the appropriate treatment. Hence, the medical

profession seems to use these words in the proper sense. The analogy, of course, is that we should

think of the program, as we begin testing it, as the sick patient.

A second problem with such definitions as “testing is the process of demonstrating that errors are

not present” is that such a goal is impossible to achieve for virtually all programs, even trivial

programs.

A third problem with the common definitions such as “testing is the process of demonstrating that

a program does what it is supposed to do” is that programs that do what they are supposed to do

still can contain errors.

The Economics of Testing

This fundamental problem will, in turn, have implications for the economics of testing,

assumptions that the tester will have to make about the program, and the manner in which test

cases are designed.

Two of the most prevalent strategies include black-box testing and white-box testing.

Black-Box Testing

Black box testing is a software testing techniques in which functionality of the software under test

(SUT) is tested without looking at the internal code structure, implementation details and

knowledge of internal paths of the software. This type of testing is based entirely on the software

requirements and specifications.

The above Black-Box can be any software system you want to test. For example: an operating

system like Windows, a website like Google, a database like Oracle or even your own custom

application. Under Black Box Testing, you can test these applications by just focusing on the inputs

and outputs without knowing their internal code implementation.

Types of Black Box Testing

There are many types of Black Box Testing but following are the prominent ones -

Functional testing - This black box testing type is related to functional requirements of a system;

it is done by software testers.

Non-functional testing - This type of black box testing is not related to testing of a specific

functionality, but non-functional requirements such as performance, scalability, usability.

Regression testing - Regression Testing is done after code fixes, upgrades or any other system

maintenance to check the new code has not affected the existing code.

White-Box Testing

White Box Testing is the testing of a software solution's internal coding and infrastructure. It

focuses primarily on strengthening security, the flow of inputs and outputs through the

application, and improving design and usability. White box testing is also known as Clear Box

testing, Open Box testing, Structural testing, Transparent Box testing, Code-Based testing, and

Glass Box testing.

It is one of two parts of the "box testing" approach of software testing. Its counter-part, blackbox

testing, involves testing from an external or end-user type perspective. On the other hand,

Whitebox testing is based on the inner workings of an application and revolves around internal

testing.

Advantages of White Box Testing

➢ Code optimization by finding hidden errors.

➢ White box tests cases can be easily automated.

➢ Testing is more thorough as all code paths are usually covered.

➢ Testing can start early in SDLC even if GUI is not available.

Disadvantages of White Box Testing

➢ White box testing can be quite complex and expensive.

➢ Developers who usually execute white box test cases detest it. The white box testing by

developers is not detailed can lead to production errors.

➢ White box testing requires professional resources, with a detailed understanding of

programming and implementation.

➢ White-box testing is time-consuming, bigger programming applications take the time to

test fully.

Software Testing Principles

Software testing is a process of executing a program with the aim of finding the error. To

make our software perform well it should be error free. If testing is done successfully it will remove

all the errors from the software.

There are seven principles in software testing:

1. Testing shows presence of defects

2. Exhaustive testing is not possible

3. Early testing

4. Defect clustering

5. Pesticide paradox

6. Testing is context dependent

7. Absence of errors fallacy

Testing shows presence of defects:

The goal of software testing is to make the software fail. Software testing reduces the

presence of defects. Software testing talks about the presence of defects and doesn’t talk about the

absence of defects. Software testing can ensure that defects are present but it cannot prove that

software is defects free. Even multiple testing can never ensure that software is 100% bug-free.

Testing can reduce the number of defects but not removes all defects.

Every application or product is released into production after a sufficient amount of testing

by different teams or passes through different phases like System Integration Testing, User

Acceptance Testing, and Beta Testing etc. So have you ever seen or heard from any of the testing

team that they have tested the software fully and there is no defect in the software? Instead of that,

every testing team confirms that the software meets all business requirements and it is functioning

as per the needs of the end user.

Example

Consider a Banking application, this application is thoroughly tested and undergoes

different phases of testing like SIT, UAT etc. and currently no defects are identified in the system.

However, there might be a possibility that in the production environment, the actual customer tries

a functionality which is rarely used in the banking system and the testers overlooked that

functionality, hence no defect was found till date or the code has never been touched by developers.

Exhaustive testing is not possible:

It is the process of testing the functionality of a software in all possible inputs (valid or

invalid) and pre-conditions is known as exhaustive testing. Exhaustive testing is impossible means

the software can never test at every test cases. It can test only some test cases and assume that

software is correct and it will produce the correct output in every test cases. If the software will

test every test cases then it will take more cost, effort, etc. and which is impractical.

It is not possible to test all the functionalities with all valid and invalid combinations of

input data during actual testing. Instead of this approach, testing of a few combinations is

considered based on priority using different techniques.

Exhaustive testing will take unlimited efforts and most of those efforts are ineffective.

Also, the project timelines will not allow testing of so many number of combinations. Hence it is

recommended to test input data using different methods like Equivalence Partitioning and

Boundary Value Analysis.

Example :

If suppose we have an input field which accepts alphabets, special characters, and numbers from

0 to 1000 only. Imagine how many combinations would appear for testing, it is not possible to test

all combinations for each input type.

Early Testing

To find the defect in the software, early test activity shall be started. The defect detected in

early phases of SDLC will very less expensive. For better performance of software, software

testing will start at initial phase i.e. testing will perform at the requirement analysis phase. Testers

need to get involved at an early stage of the Software Development Life Cycle (SDLC). Thus the

defects during the requirement analysis phase or any documentation defects can be identified. The

cost involved in fixing such defects is very less when compared to those that are found during the

later stages of testing.

Defect clustering

During testing, it may happen that most of the defects found are related to a small number of

modules. There might be multiple reasons for this like the modules may be complex, coding related

to such modules may be complicated etc.

This is the Pareto Principle of software testing where 80% of the problems are found in 20% of

the modules. We will learn more about Defect clustering and Pareto Principle later in this article.

Example : Let's consider the below image which is tested for one of the banking application and

it shows that most of the defects are related to the “Overdraft” functionality. Rest of the

functionalities like Account Summary, Funds Transfer, Standing Instruction etc., have limited

number of defects.

Pesticide paradox

Pesticide Paradox principle says that if the same set of test cases are executed again and again over

the period of time then these set of tests are not capable enough to identify new defects in the

system.

In order to overcome this “Pesticide Paradox”, the set of test cases needs to be regularly reviewed

and revised. If required a new set of test cases can be added and the existing test cases can be

deleted if they are not able to find any more defects from the system.

Repeating the same test cases again and again will not find new bugs. So it is necessary to review

the test cases and add or update test cases to find new bugs.

Testing is context dependent

Testing approach depends on context of software developed. Different types of software need to

perform different types of testing. For example, The testing of the e-commerce site is different

from the testing of the Android application.

There are several domains available in the market like Banking, Insurance, Medical, Travel,

Advertisement etc and each domain has a number of applications. Also for each domain, their

applications have different requirements, functions, different testing purpose, risk, techniques etc.

Different domains are tested differently, thus testing is purely based on the context of the domain

or application.

For Example, testing a banking application is different than testing any e-commerce or advertising

application. The risk associated with each type of application is different, thus it is not effective to

use the same method, technique, and testing type to test all types of application.

Absence of errors fallacy

If a built software is 99% bug-free but it does not follow the user requirement then it is unusable.

It is not only necessary that software is 99% bug-free but it also mandatory to fulfill all the

customer requirements.

If the software is tested fully and if no defects are found before release, then we can say that the

software is 99% defect free. But what if this software is tested against wrong requirements? In

such cases, even finding defects and fixing them on time would not help as testing is performed

on wrong requirements which are not as per needs of the end user.

For Example, suppose the application is related to an e-commerce site and the requirements

against “Shopping Cart or Shopping Basket” functionality which is wrongly interpreted and tested.

Here, even finding more defects does not help to move the application into the next phase or in the

production environment.

Inspections and Walkthroughs

The two primary human testing methods are code inspections and walkthroughs. Since the two

methods have a lot in common, we will discuss their similarities together here. Their differences

are discussed in subsequent sections.

Inspections and walkthroughs involve a team of people reading or visually inspecting a program.

With either method, participants must conduct some preparatory work. The objective of the

meeting is to find errors but not to find solutions to the errors. That is, to test, not debug.

Code inspections and walkthroughs have been widely used forsome time.In a walkthrough, a

group of developers—with three or four being an optimal number—performs the review. Only

one of the participants is the author of the program. Therefore, the majority of program testing is

conducted by people other than the author, which follows the testing principle stating that an

individual is usually ineffective in testing his or her own program.

Another advantage of walkthroughs, resulting in lower debugging (error-correction) costs, is the

fact that when an error is found it is usually precisely located in the code. In addition, this

process frequently exposes a batch of errors, allowing the errors to be corrected later en masse.

Computer-based testing, on the other hand, normally exposes only a symptom of the error (the

program does not terminate or the program prints a meaningless result), and errors are usually

detected and corrected one by one.

Code Inspections

A code inspection is a set of procedures and error-detection techniques for group code reading.

Most discussions of code inspections focus on the procedures, forms to be filled out, and so on;

here, after a short summary of the general procedure, we will focus on the actual error detection

techniques.

The moderator is expected to be a competent programmer, but he or she is not the author of the

program and need not be acquainted with the details of the program.

✓ Distributing materials for, and scheduling, the inspection session

✓ Leading the session

✓ Recording all errors found

✓ Ensuring that the errors are subsequently corrected

The moderator is like a quality-control engineer. The second team member is the programmer.

The remaining team members usually are the program’s designer (if different from the

programmer) and a test specialist.

An Error Checklist for Inspections

An important part of the inspection process is the use of a checklist to examine the program for

common errors. Unfortunately, some checklists concentrate more on issues of style than on errors

(for example, “Are comments accurate and meaningful?” and “Are ifelse, code blocks, and do-

while groups aligned?”), and the error checks are too nebulous to be useful (such as “Does the

code meet the design requirements?”). The checklist in this section was compiled after many years

of study of software errors.

Data Reference Errors

✓ Does a referenced variable have a value that is unset or uninitialized? This probably is the

most frequent programming error; it occurs in a wide variety of circumstances. For each

reference to a data item (variable, array element, field in a structure), attempt to “prove”

informally that the item has a value at that point.

✓ For all array references, is each subscript value within the defined bounds of the

corresponding dimension?

✓ For all array references, does each subscript have an integer value? This is not necessarily

an error in all languages, but it is a dangerous practice

✓ For all references through pointer or reference variables, is the referenced memory

currently allocated? This is known as the “dangling reference” problem.

✓ Does a variable’s value have a type or attribute other than what the compiler expects? This

situation might occur where a C, C++, or COBOL program reads a record into memory

and references it by using a structure, but the physical representation of the record differs

from the structure definition.

Data Declaration Errors

All variables been explicitly declared? A failure to do so is not necessarily an error, but it is a

common source of trouble.

Where a variable is initialized in a declarative statement, is it properly initialized? In many

languages, initialization of arrays and strings is somewhat complicated and, hence, error prone.

Each variable assigned the correct length and datatype?

The initialization of a variable consistent with its memory type?

Any variables with similar names (VOLT and VOLTS, for example)? This is not necessarily an

error, but it should be seen as a warning that the names may have been confused somewhere within

the program.

Computation Errors

Computations using variables having inconsistent (such as nonarithmetic) datatypes?

Any mixed-mode computations? An example is the addition of a floating-point variable to an

integer variable. Such occurrences are not necessarily errors, but they should be explored carefully

to ensure that the language’s conversion rules are understood.

Eg:

int x = 1;

int y = 2;

int z = 0;

z = x/y;

System.out.println ("z = " + z);

OUTPUT:

z = 0

Any computations using variables having the samedatatype but different lengths?

The datatype of the target variable of an assignment smaller than the datatype or result of the right-

hand expression?

An overflow or underflow expression possible during the computation of an expression

It possible for the divisor in a division operation to be zero

Comparison Errors

Comparisons between variables having different datatypes, such as comparing a character string

to an address, date, or number.

Comparison operatorssuch relations as at most, at least, greater than, not less than, less than or

equal.

Programmers often make mistakes when writing logical expressions involving and, or, and not.

For example:

int x=1;

int y=2;

int z=3;

if(x>y)&&(x>z)

System.out.println ("x = " + x);

else if(y>z)

System.out.println ("y = " + y);

else

System.out.println ("z = " + z);

Output

z=3

Control-Flow Errors

If the program contains a multiway branch such as a computed GO TO, can the index variable ever

exceed the numberof branch possibilities? For example, in the statement GO TO (200,300,400)

where i will always have the value of 0,1,2, or 3?

Every loop eventually terminate? Devise an informal proof or argument showing that each loop

will terminate.

This is a common error in zero-based loops. For example, if you want to create Java code for a

loop that counted to 10, the following would be wrong, as it counts to 11:

for (int i=0; i<=10;i++)

{

System.out.println(i);

}

Interface Errors

The number of parameters received by this module equals the number of arguments sent by each

of the callingmodules.

The attributes (e.g., datatype and size) of each parameter match the attributes of each

corresponding argument.

If built-in functions are invoked, are the number, attributes, and order of the arguments correct?

If a module or class has multiple entry points, is a parameter ever referenced that is not associated

with the current point of entry.

A: PROCEDURE(W,X);

W=X+1;

RETURN

B: ENTRY (Y,Z);

Y=X+Z;

END;

Does a subroutine alter a parameter that is intended to be only an input values

Global variables are present, do they have the same definition and attributes in all modules that

reference them.

Input/Output Errors

✓ Files are explicitly declared, are their attributes correct.

✓ the attributes on the file’s OPEN statement correct.

✓ Sufficient memory available to hold the file your program will read.

✓ Have all files been opened before use.

✓ Have all files been closed after use.

✓ Are end-of-file conditions detected and handled correctly.

✓ Are I/O error conditions handled correctly.

✓ Are there spelling or grammatical errors in any text that is printed or displayed by the

program.

White-Box Testing

White Box Testing is the testing of a software solution's internal coding and infrastructure. It

focuses primarily on strengthening security, the flow of inputs and outputs through the application,

and improving design and usability. White box testing is also known as Clear Box testing, Open

Box testing, Structural testing, Transparent Box testing, Code-Based testing, and Glass Box

testing.

Logic-Coverage Testing

Logic corresponds to the internal structure of the code and this testing is adopted for safety-

critical applications such as software’s used in aviation industry. This Test verifies the subset of

the total number of truth assignments to the expressions.

Logic Coverage Sources:

Logic coverage comes from any of the below mentioned sources:

✓ Decisions in programs

✓ Requirements

The equivalent Java code snippet follows:

public void foo(int a, int b, int x) {

if (a>1 && b==0) {

x=x/a;

}

if (a==2 || x>1) {

x=x+1;

}

}

You could execute every statement by writing a single test case that traverses path ace. That is,

by setting A=2, B=0, and X=3 at point a, every statement would be executed once (actually, X

could be assigned any value).

Exceptions in Login-coverage testing:

✓ Programs with no decisions.

✓ Programs or subroutines/methods with multiple entry points. A given statement might be

executed only if the program is entered at a particular entry point.

✓ Statements within ON-units. Traversing every branch direction will not necessarily cause

all ON-units to be executed.

Equivalence partitioning:

Equivalence partitioning (EP) is a specification-based or black-box technique.

It can be applied at any level of testing and is often a good technique to use first.

The idea behind this technique is to divide (i.e. to partition) a set of test conditions into groups or

sets that can be considered the same (i.e. the system should handle them equivalently), hence

‘equivalence partitioning’. Equivalence partitions are also known as equivalence classes

For example, a savings account in a bank has a different rate of interest depending on the balance

in the account. In order to test the software that calculates the interest due, we can identify the

ranges of balance values that earn the different rates of interest. For example, 3% rate of interest

is given if the balance in the account is in the range of $0 to $100, 5% rate of interest is given if

the balance in the account is in the range of $100 to $1000, and 7% rate of interest is given if the

balance in the account is $1000 and above, we would initially identify three valid equivalence

partitions and one invalid partition as shown below.

In the above example we have identified four partitions, even though the specification mentioned

only three. This shows a very important task of the tester that is a tester should not only test what

is in the specification, but should also think about things that haven’t been specified. In this case

we have thought of the situation where the balance is less than zero.

Boundary value analysis

Boundary value analysis is a test case design technique to test boundary value between partitions

(both valid boundary partition and invalid boundary partition). A boundary value is an input or

output value on the border of an equivalence partition, includes minimum and maximum values at

inside and outside boundaries. Normally Boundary value analysis is part of stress and negative

testing.

Using Boundary Value Analysis technique tester creates test cases for required input field. For

example; an Address text box which allows maximum 500 characters. So, writing test cases for

each character once will be very difficult so that will choose boundary value analysis.

Example for Boundary Value Analysis:

Suppose you have very important tool at office, accepts valid User Name and Password field to

work on that tool, and accepts minimum 8 characters and maximum 12 characters. Valid range 8-

12, Invalid range 7 or less than 7 and Invalid range 13 or more than 13.

✓ Test Cases 1: Consider password length less than 8.

✓ Test Cases 2: Consider password of length exactly 8.

✓ Test Cases 3: Consider password of length between 9 and 11.

✓ Test Cases 4: Consider password of length exactly 12.

✓ Test Cases 5: Consider password of length more than 12.

Cause Effect Graphing

Cause-Effect Graph graphically shows the connection between a given outcome and all issues

that manipulate the outcome. Cause Effect Graph is a black box testing technique.

It is generally uses for hardware testing but now adapted to software testing, usually tests

external behavior of a system. It is a testing technique that aids in choosing test cases that

logically relate Causes (inputs) to Effects (outputs) to produce test cases.

Uses of Cause effect graphing

✓ To determine the current problem so that right decision can be taken very fast.

✓ To narrate the connections of the system with the factors affecting a particular process or

effect.

✓ To recognize the probable root causes, the cause for a exact effect, problem, or outcome.

Error Guessing

Any particular methodology such as boundary-value analysis of cause-effect graphing, these

people seem to have a knack for sniffing out errors.test-case-design technique that could be

termed error guessing.

The basic idea is to enumerate a list of possible errors or error-prone situations and then write

test cases based on the list.

✓ The input list is empty.

✓ The input list contains one entry.

✓ All entries in the input list have the same value.

✓ The input list is already sorted.

Consider the MTEST program in the section on boundary-value analysis using the error-

guessing technique:

✓ Does the program accept “blank” as an answer?

✓ A type-2 (answer) record appears in the set of type-3 (student) records.

✓ A record without a 2 or 3 in the last column appears as other than the initial (title) record.

✓ Two students have the same name or number.

✓ Since a median is computed differently depending on whether there is an odd or an even

number of items, test the program for an even number of students and an odd number of

students.

✓ The number-of-questions field has a negative value.

Compare and contrast between black box testing and white box testing

Black box testing White box testing

It is a way of software testing in which the

internal structure or the program or the code is

hidden and nothing is known about it.

It is a way of testing the software in which the

tester has knowledge about the internal structure

the code or the program of the software.

It is mostly done by software testers. It is mostly done by software developers.

No knowledge of implementation is needed. Knowledge of implementation is required.

It can be referred as outer or external software

testing. It is the inner or the internal software testing.

It is functional test of the software. It is structural test of the software.

This testing can be initiated on the basis of

requirement specifications document.

This type of testing of software is started after

detail design document.

No knowledge of programming is required. It is mandatory to have knowledge of

programming.

It is the behavior testing of the software. It is the logic testing of the software.

It is applicable to the higher levels of testing of

software.

It is generally applicable to the lower levels of

software testing.

It is also called closed testing. It is also called as clear box testing.

It is least time consuming. It is most time consuming.

It is not suitable or preferred for algorithm

testing. It is suitable for algorithm testing.

Can be done by trial and error ways and

methods.

Data domains along with inner or internal

boundaries can be better tested.

Example: search something on google by using

keywords Example: by input to check and verify loops

Types of Black Box Testing:

A. Functional Testing

B. Non-functional testing

C. Regression Testing

Types of White Box Testing:

A. Path Testing

B. Loop Testing

C. Condition testing

Blackbox testing

Black box testing is defined as a testing technique in which functionality of the Application Under

Test (AUT) is tested without looking at the internal code structure, implementation details and

knowledge of internal paths of the software. This type of testing is based entirely on software

requirements and specifications.

In BlackBox Testing we just focus on inputs and output of the software system without bothering

about internal knowledge of the software program.

The above Black-Box can be any software system you want to test. For Example, an operating

system like Windows, a website like Google, a database like Oracle or even your own custom

application. Under Black Box Testing, you can test these applications by just focusing on the

inputs and outputs without knowing their internal code implementation.

How to do BlackBox Testing

Here are the generic steps followed to carry out any type of Black Box Testing.

• Initially, the requirements and specifications of the system are examined.

• Tester chooses valid inputs (positive test scenario) to check whether SUT processes them

correctly. Also, some invalid inputs (negative test scenario) are chosen to verify that the

SUT is able to detect them.

• Tester determines expected outputs for all those inputs.

• Software tester constructs test cases with the selected inputs.

• The test cases are executed.

• Software tester compares the actual outputs with the expected outputs.

• Defects if any are fixed and re-tested.

Types of Black Box Testing

There are many types of Black Box Testing but the following are the prominent ones -

Functional testing - This black box testing type is related to the functional requirements of a

system; it is done by software testers.

Non-functional testing - This type of black box testing is not related to testing of specific

functionality, but non-functional requirements such as performance, scalability, usability.

Regression testing - Regression Testing is done after code fixes, upgrades or any other system

maintenance to check the new code has not affected the existing code.

Tools used for Black Box Testing:

Tools used for Black box testing largely depends on the type of black box testing you are doing.

• For Functional/ Regression Tests you can use - QTP, Selenium

• For Non-Functional Tests, you can use - LoadRunner, Jmeter

Black Box Testing Techniques

Following are the prominent Test Strategy amongst the many used in Black box Testing

Equivalence Class Testing: It is used to minimize the number of possible test cases to an

optimum level while maintains reasonable test coverage.

Boundary Value Testing: Boundary value testing is focused on the values at boundaries. This

technique determines whether a certain range of values are acceptable by the system or not. It is

very useful in reducing the number of test cases. It is most suitable for the systems where an

input is within certain ranges.

Decision Table Testing: A decision table puts causes and their effects in a matrix. There is a

unique combination in each column.