Software Engineering - chp7- tests

MedTech

Chapter 7 – Tests

Types of Tests, Verification, Methods…

Dr. L i l ia SFAXIwww. l i l iasfax i .wix .com /l i l iasfax i

S l id e 1

MedTech – Mediterranean Institute of TechnologyCS321-Software Engineering

MedTech

MedTech

Why?

Dr. L i l ia SFAXI www. l i l iasfax i .wix .com /l i l iasfax i

S l id e 2

Software Testing

MedTech

Why?

• Because Software is Buggy!

• Cost of bugs to the US economy 60 billion dollars per year

• Software contain in average 1-5 bugs every 1000 lines of code• Having 100% correct mass-market software is impossible

• We must verify the software as much as possible


S l id e 3

Software Testing

MedTech

Common Terms

• Official IEEE terminology• Failure

• Observable incorrect behaviour of the software

• More related to the behavior of the program rather than its code

• Fault• Also called a bug

• Incorrect code

• Necessarely but not sufficient condition of the occurrence of a failure

• Error• Cause of a fault

• Human error, conceptual…


S l id e 4

Software Testing

MedTech

Common Terms: Example

• Let’s take this method: int doubleValue(int val){

int result;result = val * val;return result;

}

• Is the fact that this method is supposed to return the double of the input but doesn’t is considered a failure, fault or error?• Failure, because it is related to the behavior of the method

• In this case, which line of code represents the fault?• result = val * val;

• And what is the error that caused this bug?• Only God and the developer know…


S l id e 5

Software Testing

MedTech

VERIFICATION APPROACHESSoftware Testing


S l id e 6

MedTech

Verification Approaches

• How can we verify our software system?

• Four main approaches:• Testing (dynamic verification)

• Static verification

• Inspections

• Formal Proofs of Correctness

• In our course, we will focus on testing, as it is the most common and used verification approach• We start first by summarizing the difference between these approaches


S l id e 7


MedTech

Testing

• Testing:• Exercising the system to try to make it fail

• Test Case: {𝑖 ∈ 𝐷, 𝑜 ∈ 𝑂}• A set of inputs in the domain D, and an expected output of the demain O

• Test Suite: Set of test cases


S l id e 8


MedTech

Static Verification

• Static Verification:• Tries to identify precise causes of problems in the program, such as null

pointer

• Instead of considering a precise input, it considers all possible inputs (executions)

• Unlike testing, the verification is complete


S l id e 9


MedTech

Inspections

• Inspections:• Also called Reviews or Walkthroughs

• Manual and group activity, where several developers look at the code in order to identify defects

• Quite effective in practice

• Used widely in industry


S l id e 10


MedTech

Formal Proof

• Formal Proof of Correctness:• Given a software specification, a formal specification is a document that

formally defines the expected behaviour of the software

• A formal proof of correctness prooves that the program being verifiedactually implements the program specification

• Is done through a sofisticated mathematical analysis of the specificationand of the code


S l id e 11


MedTech

Pros and Cons

Approach Pros Cons

Testing No False Positives Highly incomplete : testing considers a tinyfraction of the input domain

Static Verification Considers all program behaviours Considers some impossible behaviours : cangenerate false positive

Inspections SystematicThorough

Manual processInformalSubjective: depends on the people

Formal Proofs Strong guarantees Need a formal specification, rarely availableComplex and expensiveNeeds very a specialized personal


S l id e 12


MedTech

TESTING APPROACHSoftware Testing


S l id e 13

MedTech

Definition of Testing

• Today, Quality Assurance (Verification) is mostly about Testing• « 50% of my company employees are testers, and the rest spend 50% of

their time testing » Bill Gates• Testing

• Executing a program on a (tiny) sample of the input domain• Dynamic technique: the program has to be executed in order to test• Optimistic approximation:

• Program under test being performed on a small subset of the input domain, it istherefore supposed that the behavior of any other input is consistant with thatshown with the selected input data

• Successful tests:• « A test is successful if the program fails » Goodenough and Gerhart• Testing cannot prove the absence of error, but can only prove their presence• If all tests are passed

• Either the program is correct (unlikely)• Or the set of tests is bad (more l ikely)


S l id e 14

Testing Approach

MedTech

Testing Granularity Levels


S l id e 15


MedTech


• Unit Testing• Testing single modules one by one

• Integration Testing• Testing multiple modules• Testing their interactions

• System Testing• Testing of the complete system• Both functional and non functional requirements

• Acceptance Testing• Validation of the software against the customer requirements

• Regression Testing• Done when the system changes• Verifies that the changes did not impact the behaviour of the system


S l id e 16


MedTech

Alpha and Beta Testing

• All previous testing levels are done at the level of the developingorganization, by the developers themselves

• Alpha Testing• A form of internal acceptance testing performed mainly by inhouse software

QA and testing teams

• Last testing done by test teams at the development site• After acceptance testing

• Before releasing the software for beta tests

• Beta Testing• Final testing phase

• Testing carried out by real users in real environement• Generally more accurate because the customers are less tolerant for

problems


S l id e 17

Testing Types

MedTech

TESTING TECHNIQUESSoftware Testing


S l id e 18

MedTech

Black-Box and White-Box Testing


S l id e 19

Testing Techniques

MedTech

Black-Box Testing

• A kind of testing in which we consider the software as a closed box

• Based on the description of the software (specification)• We don’t inspect the code of the software

• Objective:• Cover as much specified behavior as possible

• Limitations• Cannot reveal errors due to implementation details

• Advantages• Focuses on the domain of the software, and is used to make sure that it is

covering the important behaviors• No need for the code, can be used for legacy systems• It is possible to start test design early• Applicable at all granularity levels


S l id e 20

Black-Box Testing

MedTech

Black-Box Testing: Example

• Specification: The function inputs an integer and prints it• Code

void printNumBytes (int param){if (param <1024) printf ("%d", param);else printf ("%d Kb",param/124);

}

• If we look only at the specification, we would perform at least 3 tests: with a positive, negative and null param• The result would be conform to the specification, if the tested integer is

less than 1024

• Black box testing would not show the error at line 3 (124 instead of 1024)


S l id e 21

Black-Box Testing

123

MedTech

Systematic Functional Testing Approach


S l id e 22

Black-Box Testing

MedTech

Step1: Identify Independently Testable Features

• Identifying all functionalities of the software that can be testedindependently from the others

• Example• For a simple function, there is only one testable feature

• For a complex application, like a spreadsheet, we can have multiple features:• Merge cells

• Statistical function

• Chart creation

• …


S l id e 23

Black-Box Testing

MedTech

Step2: Select Test Data

• Identifying the relevant inputs for each function defined earlier• Consider the following case:

• How can we select inputs to run our tests?

• What are the relevant inputs, those that, once tested, can make us confident that the software is behaving correctly?


S l id e 24

Black-Box Testing

MedTech

Step2: Select Test Data: Selection Types

• Exhaustive Testing: Test ALL possible inputs• Quite impossible in some cases, even with powerful software

• For example, for the function printSun (int a, int b), if an integer’s size is32 bits, if the system runs 1 test per nanosecond, it would take us 600 years to test all possible combinations!

• Random Testing: Pick the inputs randomly in the domain• Has advantages

• Inputs are picked uniformly: all inputs are considered equal

• Avoids designer bias

• Problem• Bugs are mostly scarse, so picking the inputs randomly will likely miss them


S l id e 25

Black-Box Testing

MedTech

Step2: Select Test Data: Selection Types

• Partition Testing: Divide the input domaininto partitions• Partitions : areas of the domain that are

considered homogenuously by the software

• Idea: Bugs tend to be dense in somepartitions

• Objective: • Identify partitions in the input domain• Select inputs from each partition

• The parameters must be treatedindependenly to pick different partitions• Select Boundary Values in partitions


S l id e 26

Black-Box Testing

MedTech

Step3: Test Cases Specifications

• Test Cases Specifications• Define how the values should be put together when actually testing the

system

• Combine the possible

inputs, using a cartesian

product, for example

• Eliminate meaninglesscombinations


S l id e 27

Black-Box Testing

MedTech

Model-Based Testing


S l id e 28

Black-Box Testing

MedTech

Model-Based Testing


S l id e 29

Black-Box Testing

• To go from the specification to test cases, we have to construct a model• An abstract representation of the software under test

• We can use many possible models

• FSM: Finite State Machine• A possible model

MedTech

Model-Based Testing


S l id e 30

Black-Box Testing

• To go from the specification to test cases, we have to construct a model• An abstract representation of the software under test

• We can use many possible models

• FSM: Finite State Machine• A possible model

MedTech

Model-Based Testing: Steps


S l id e 31

Black-Box Testing

MedTech

Model-Based Testing: Example

1. Start with an informal specification


S l id e 32

Black-Box Testing

MedTech


2. Build a model from the specification


S l id e 33

Black-Box Testing

MedTech


3. Extract Test Cases fromthe model by trying to• Cover all the states

• Cover all the transitions


S l id e 34

Black-Box Testing

MedTech

White-Box Testing

• Complementary to the black-box testing• Based on the code• Assumption:

• Executing the faulty statement is a necessary condition for revealing a fault

• Objective• Cover as much coded behavior as possible

• Limitations• Cannot reveal errors due to missing paths: parts of the software

specification that are not implemented• Because it is focused on the existing code, not on the specification

• Advantages• The quality of the white-box testing can be measured objectively• The test can be measured automatically


S l id e 35

White-Box Testing

MedTech

White-Box Testing: Example

• Specification: The function inputs an integer parameter and returnshalf of its value if even, its value otherwise

• Codeint fun (int param){

int result = param/2;return result;

}

• If we look only at the code, we guess that the function is supposed to return the half of any number given, which is correctly done.• With white box testing only, we wouldn’t know that we have to test even

and odd numbers, and that each one gives a different output.


S l id e 36

White-Box Testing

123

MedTech

Test Coverage

• Assumption• Executing the faulty statement is a necessary condition for revealing a

fault

• So, in order for a test to have a chance to detect most of the bugs, itmust « cover » a big percentage of the code.

• Code coverage:• A measure (percentage) used to describe the degree to which the source

code of a program is executed when a particular test suite runs

• A program with high code coverage has more of its source code executedduring testing, which suggests it has a lower chance of containingundetected software bugs compared to a program with low code coverage.


S l id e 37

White-Box Testing

MedTech

Test Coverage Criteria

• Function Coverage• Has each function (or subroutine) in the program been called?

• Statement Coverage• Has each statement in the program been executed?

• Branch Coverage• Has each branch (also called DD-path (Decision to Decision Path)) of each

control structure (such as in if and case statements) been executed?

• Condition Coverage• Has each Boolean sub-expression evaluated both to true and false?


S l id e 38

White-Box Testing

MedTech

Test Coverage Criteria

• Coverage Criteria are defined in terms of :• Test Requirements

• Elements, Entities in the code that we need to execute, in order to satisfy the criteria

• Result: • Test Specifications

• Descriptions of how the test should be in order to satisfy the requirements

• Test Cases• Instanciations of test specifications


S l id e 39

White-Box Testing

MedTech

Test Coverage Criteria: Example

printSum (int a, int b){int result = a+b;if (result>0)

printcol("red", result);else if (result<0)

printcol("blue", result);}

Test Specifications: conditions on a and b in order to cover all test requirementsTest Spec #1: a+b > 0Test Spec #2: a+b < b

Examples of Test Cases:#1: ((a=[3], b=[9]), (output color=[red], output value=[12]))#2: ((a=[-5], b=[-8]), (output color=[blue], output value=[-13]))


S l id e 40

White-Box Testing

Test Requirement 1

Test Requirement 2

MedTech

Statement Coverage

• Basic and simplest coverage criteria• Test Requirements

• All the statements in the program

• Coverage measure

• *+,-./01.2.3+4.56474.,.*4680479*+,-./016474.,.*46




With TC #1, almost 71% of the statements are covered (st #1,2,3,4,7)

When adding TC #2, 100% are covered!


S l id e 41

White-Box Testing

TC #1a=3b=9

TC #2a=-5b=-8

1234567

MedTech

Control-Flow Graph

• Convenient representation of the code when you want to reason about its structure





S l id e 42

White-Box Testing

1

2

3

5 4

6

7

TF

T

F

1234567

MedTech

Branch Coverage

• Branch: Outgoing edges from a decision point (if, loop…)• Test Requirements

• Branches in the program• Coverage Measure

• *+,-./01.2.3+4.5-/7*3:.680479 *+,-./01-/7*3:.6




TC #1 covers branch #1TC #2 covers branch #2Branch #3 is never covered!

=> We add a new TC


S l id e 43

White-Box Testing

1

2

3

5 4

6

7

TF

T

F

TC #1a=3b=9

TC #2a=-5b=-8

TC #3a = 0b = 0

1234567

MedTech

Condition Coverage

• Test Requirements• Individual conditions in the program

• Coverage Measure

• *+,-./0130*5;4;0*64:747/.-04:87*5<80479*+,-./0130*5;4;0*6

void main(){float x,y;read(x);read(y);if ((x==0)||(y>0))

y = y/x;else x = y+2;write(x);write(y);

}


S l id e 44

White-Box Testing

3

4

7 6

8

x==0||y>01

2345678910

9

!(x==0||y>0)

MedTech

Condition Coverage



}

Branch coverage = 100%

But Condition coverage is not complete, because we don’thave a test case where x==0 is true


S l id e 45

White-Box Testing

3

4

7 6

8

x==0||y>0

12345678910

9

!(x==0||y>0)

TC #1x=5y=5

TC #2x=5y=-5

MedTech

Condition Coverage



}

Condition coverage = 100% because with both TC we covered

all the possible values for both conditionsBut Branch coverage is not complete.


S l id e 46

White-Box Testing

3

4

7 6

8

x==0||y>0

12345678910

9

!(x==0||y>0)

TC #1x=5y=5

TC #3x= 0y=-5

MedTech

Test Criteria Subsumption

• We say that Branch Coverage subsumes Statement Coverage• If a test suite satisfies branch coverage, it satisfies statement coverage

too

• But Branch Coverage doesn’t subsume Condition Coverage and Condition Coverage doesn’t subsume Branch Coverage


S l id e 47

White-Box Testing

Statement Coverage

Branch Coverage

Condition Coverage

MedTech

Branch and Condition Coverage

• Also called Decision and Condition Coverage

• Considers both Branch Coverage and Condition Coverage

• Test Requirements• Branches and individual conditions in the

program

• Coverage Measure• Both coverage measures


S l id e 48

White-Box Testing

Statement Coverage

Branch Coverage

Condition Coverage


MedTech




}

Condition coverage = 100% Branch coverage = 100%


S l id e 49

White-Box Testing

3

4

7 6

8

x==0||y>0

12345678910

9

!(x==0||y>0)

TC #1x=5y=5

TC #3x= 0y=-5

TC #2x=5y=-5

MedTech

Other Criteria

• MC/DC: Modified Condition/Decision Coverage• Required for safety critical applications• Test important combinations of conditions instead of all of them• Extends branch and decision coverage with the requirement that each

condition should affect the decision outcome independently

• Path Coverage• Test requirements: all the paths in the program must be covered• Incredibly expensive, because we have virutally infinite number of TC

• Data Flow Coverage• Coverage of statements where a memory location is modified and

statements where this memory location is used

• Mutation Coverage• Evaluation of the quality of the test suite by modifying slightly the

statements in the code to see if the tests detect them


S l id e 50

White-Box Testing

MedTech

Test Criteria Subsumption


S l id e 51

White-Box Testing

Statement Coverage

Branch Coverage

Condition Coverage


MC/DC

Multiple Condition Coverage

Path CoverageMutation Coverage

Data-Flow Coverage

Theoritical Criteria

Practical Criteria

MedTech

NON FUNCTIONAL TESTINGSoftware Testing


S l id e 52

MedTech

Non Functional Testing

• Testing an application from its non-functional attributes• Designed to figure out if your product will provide a good user

experience

• Types of non-functional testing:• Performance Testing

• Load Testing

• Stress Testing• Usability Testing

• Security Testing

• Portability Testing


S l id e 53


MedTech

Performance Testing

• Mostly used to identify any bottlenecks or performance issues rather thanfinding bugs in a software

• One of the most important and mandatory testing types• Determines how a system performs in terms of responsiveness and stability

under a particular workload• Tests:

• Speed (response time, data rendering and access..)• Capacity, stability, scalability

• Performance testing• Can be either qualitative or quantitative • Can be divided into different sub-types such as Load testing and Stress

testing.• Unless tests are conducted on the production hardware, from the same

machines the users will be using, there will always be a degree of uncertainty in the results.


S l id e 54


MedTech

Load Testing

• Verifies application behavior under normal and peak load conditions• Measures

• Response times

• Throughput rates

• Resource-utilization levels

• Identifies your application’s breaking point, assuming that the breaking point occurs below the peak load condition

• Helps to determine• How many users the application can handle before performance is

compromised.

• How much load the hardware can handle before resource utilization limitsare exceeded.


S l id e 55


MedTech

Stress Testing

• When the load placed on the system is raised beyond normal usage patterns to test the system's response at unusually high or peak loads

• Reveal application bugs that surface only under high load conditions• Synchronization issues

• Race conditions• Memory leak

• Provides an estimate of how far beyond the target load an application can go before causing failures and errors in addition to slowness.

• Allows you to establish application-monitoring triggers to warn of impending failures.

• Ensures that security vulnerabilities are not opened up by stressfulconditions.


S l id e 56


MedTech

Usability Testing

• A black-box technique • Used to identify any error(s) and improvements in the software by

observing the users through their usage and operation• Used in user-centered interaction design• Evaluates a product by testing it on users• Can be defined in terms of five factors

• Efficiency of use• Learn-ability• Memory-ability• Errors/safety• Satisfaction

• Usually involves systematic observation under controlled conditions to determine how well people can use the product

• Focuses more on the UX (User Experience) than the UI (User Interface)


S l id e 57


MedTech

Security Testing

• Tests if the application is vulnerable to attacks• Main security requirements

• Confidentiality

• Integrity

• Authentication• Availability

• Authorization

• Non-repudiation.

• In order to define the main security issues and possible mechanismsfor your system, you can consult the Open Web Application Security Project (OWASP: https://www.owasp.org)


S l id e 58


MedTech

TESTING DOCUMENTATIONSoftware Testing


S l id e 59

MedTech

Documentation

• Documentation of artifacts that should be developed before or duringthe testing of Software

• Documentation for software testing helps in estimating:• Required testing effort

• Test coverage• Requirement tracking/tracing, etc.

• Commonly used documented artifacts related to software testing :• Test Plan

• Test Scenario• Test Case

• Traceability Matrix


S l id e 60

Documentation

MedTech

Test Plan

• Outlines:• The strategy that wil l be used to test an application• The resources that wil l be used• The test environment in which testing wil l be performed• The l imitations of the testing and the schedule of testing activities

• Responsible: Quality Assurance Team Lead • Includes:

• Introduction to the Test Plan document• Assumptions while testing the application• List of test cases included in testing the application• List of features to be tested• What sort of approach to use while testing the software• List of deliverables that need to be tested• The resources allocated for testing the application• Any risks involved during the testing process• A schedule of tasks and milestones to be achieved

• See the IEEE Test Plan Outline (given in the related documentation)


S l id e 61

Documentation

MedTech

Test Scenario & Test Case

• Test Case• A set of input values, execution precondition, expected results and

executed post condition, developed to cover a certain test condition.

• Test Scenario• A detailed test procedure.

• Has many test cases associated with it.• Tends to make sure that end to end functionality of application under test

is working as expected

• Checks if the all business flows are working as expected

• The tester needs to put his/her foot in the end users shoes to check and perform the action as how they are using application under test

• The tester needs to consult or take help from the client, stakeholder or developers


S l id e 62

Documentation

MedTech

Traceability Matrix

• Table that is used to trace the requirements during the SDLC• Can be used for:

• Forward tracing (i.e. from Requirements to Design or Coding) • Backward tracing (i.e. from Coding to Requirements)

• Each requirement in the RTM document is linked with its associatedtest case

• The Bug ID is also included and linked with its associated requirementsand test case.

• Goals:• Make sure that the software is developed as per the mentioned

requirements.• Helps in finding the root cause of any bug.• Helps in tracing the developed documents during different phases of SDLC


S l id e 63

Documentation

MedTech

Traceability Matrix


S l id e 64

Documentation

MedTech

References


S l id e 65

• Software Development Process (MOOC), https://classroom.udacity.com/courses/ud805/lessons/3608718991/concepts/4404287840923#

• Software Testing, Tutorials Point, https://www.tutorialspoint.com/software_testing/software_testing_quick_guide.htm