Software Testing Chapter 12: Subsystems Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen

Software TestingChapter 12: Subsystems

Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen

2

Overview

• Subsystems What are subsystems Types of subsystems Why test them

• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test

3

Overview



4

Subsystems

• What are subsystems Any testable collection of classes, objects, components

and modules• Characteristics

Executable and testable as a whole• Whatever its size, it must be executable to be testable• Candidate for responsibility testing

Has parts that can be tested in isolation Implements a cohesive set of responsibilities Doesn’t provide all the functionality of the application

under test

5

Subsystems

• Types of subsystems Small clusters

• Several tightly coupled classes• Testing the classes in isolation sometimes impractical

(e.g. cyclic dependencies)

Large clusters• Group of related classes or clusters

Pattern participants• Implementation of design pattern tested as a subsystem (e.g.

Model-View-Controller implementation)

6

Subsystems

• Types of subsystems (continued) Build group

• Collection of software components processed by the linkage editor to generate an executable

Task group• Collection of build groups that is linked as an executable

process or that generate processes that do something interesting

Family of Ada 95 packages Developer subclasses

7

Subsystems

• Why test them It is a precondition for testing at system scope It is a practical necessity Support for internal release during incremental,

iterative development Use cases can model requirements for any subsystem

with public interface Some classes and other components have been

bundled for historical reasons and must be tested as a whole

8

Subsystems

• Why test them (continued) Some subsystems must work to receive additional

funding Testing budget or schedule is limited or reduced Dependability of some subsystems is higher than of

others The target environment is unique or different or must

be simulated apart from the rest of the application

9

Overview



10

Subsystem Test Design Pattern

• Subsystem test plan What features and capabilities should be tested? How should the test plan and test suites be organized?

• Structure and responsibilities determine testing approach

• Key problem: Developing a testable model of aggregate behavior

> Solution: test design patterns

11

Overview



12

Class Association Test

• Intent Design a test suite to verify the implementation of

required associations among classes• Context

Applicable when the SUT must correctly implement the relationships represented in a class diagram

Reveal bugs in implementation of associations

13


• Fault Model Kinds of faults

• Incorrect multiplicity Legal combination rejected Illegal combination accepted

• Update anomaly The link that associates several instances of one class with a single instance of

another class necessarily carries some redundant information

• Delete anomaly What happens to owned objects if the owning object is deleted? (depends on

design/requirements)

• Missing link Not all links are created or some links become corrupted

• Wrong link All links created but some links are incorrect or inconsistent

14


• Fault Model Faults can be reached by exercising the code that

implements the association constraint This code lies on a path that will be executed when

boundary values of the constraints are input (see Section 10.2.4, Domain Testing Model)

15


• Strategy Test model

• Multiplicity test sets Association A:B with multiplicity (e.g. 0..1, 0..n, 1..1, 1..n, …) has

2 boundary values Binary association A:B and B:A has 4 boundary values Apply 1x1 test strategy (with on points, off points and in points)

and cast results in domain matrix (see Section 10.2.4, Domain Testing Models)

• Testing for update/delete bugs• Wrong and missing links

16


• Automation Increase testing by interleaving anomaly tests with

max‐value multiplicity tests SUT can be exercised with an API driver interface GUI may provide sufficient control to exercise

associations DBMS: database loader utility to setup test cases

17


• Entry criteria Each class in the scope of the model has passed an

alpha‐omega cycle Keyed collections are typically used to implement

associations. These classes should be individually tested with Quasi‐modal Class Test

• Exit criteria Each multiplicity on point test passes Each multiplicity off point test passes A deletion anomaly test passes An update anomaly test passes All tests for mutual exclusion pass

18


• Consequences N associations at most 8n basic multiplicity test cases

(2 for each min and max boundary on A:B association and 2 for each min and max boundary on B:A association)

Typical system: several 100 associations several 1000 test cases

Class association test requires a model of the class associations in the system under test

Difficult for maximal value testing if number of database instances is large

19

Overview



20

Controlled Exception Test

• Intent Design a test suite that will exercise exception

handlers and provide a systematic way to generate all exceptions

• Fault model Safe exception handling = difficult! Several pitfalls

Improperly passed exceptions Exception is ignored and results in abrupt task end Ignored exceptions resulting in mysterious failures Looping exception structure (retry without limitation) Wrong exception is thrown/wrong catch block activated …

21


• Test Model To test properly exceptions must be triggered

Can be difficult!

22


• Test Model To test properly exceptions must be triggered

Can be difficult!

3 ways for exception testing• Exception activation: Trough input and state values

supplied by test cases• Exception inducement: By directly manipulating the

implementation under test• Exception simulation: By using controllable wrappers

for virtual machine services

23


• Automation Stubs to simulate application specific exceptions Command-line scripts to generate anomalous

conditions which (hopefully) trigger exceptions Wrapper classes

24

25


• Consequences Design of exception-handling test suite requires

technical understanding of which exceptions can be raised

• Target environment exceptions (e.g. Java Class Libraries’ exceptions)

Well documented• Applications-specific exceptions

Typically undocumented

Testing exceptions in an operational system may be difficult!

26

Overview



27

Round-trip Scenario Test

• Intent Develops path set with minimal branch & loop

coverage for UML Sequence Diagrams by extracting a control flow model

• Fault Model Lots of different round-trip scenarios

• Scenario: Path trough sequence diagram Lots of scenarios = lots of possible bugs

• Incorrect or missing output• Missing functions or features• Incorrect messages passed• Deadlocks• …

28

29


• Test Model Sequence diagrams are not ideal for testing!Transform sequence diagram to flow graph

• Flow graphs are highly testable• Node: Chunk of unconditional processing• Arrows: Show which node may follow another

30

Test Procedure – Step 1

1. Convert Sequence Diagram into Flow Graph

• Each consecutive group of messages becomes a segment (block)

• Each bracketed condition becomes a decision (hexagon)• Each bracketed loop condition also becomes a decision

(hexagon) – The position of loop predicates is important here (‘while’ & ’for’ loops vs. ‘until’ loops)

Often this conversion will already reveal some ambiguities and omissions!

31

32


2. Identify Paths to Test• No loops and branches?

» Any scenario will reach all segments• Loops or braches?

» Often astronomical number of possible paths

Running over all possible paths is in most cases infeasible!

33


Solution:• Take every branch at least once• Use every loop in at least 3 ways

• Bypass the loop (or do minimum # iterations)• Do 1 iteration• Do the maximum number of iterations

• Advanced tools are not necessary! Use colored markers to draw the paths

34


Procedure:1. Choose the longest or most complex entry-

exit path and color it2. Switch to another color and color another

entry-exit path3. Repeat step 2 until all decision branches are

colored4. Make a table with a row for each path listing

the conditions that must hold for this path

35

36


Procedure - Continued6. Pick any path that leads to the loop condition7. Copy the paths up to the loop entry in the

table and append the predicate conditions to allow one iteration, append the exit path

8. Pick another path to the loop entry and append the predicate conditions to allow the maximum number of iterations, append the exit path

37

38


Path Comment Path Conditions

1 Longest path, all false, Bypass loop !notATMCard, !isStolen, !accountClosed, validPIN, !declineFee

2 Branch 1 true, Bypass loop notATMCard

3 Branch 2 true, Bypass loop !notATMCard, isStolen

4 Branch 3 true, Bypass loop !notATMCard, !isStolen, accountClosed

5 Branch 4 true, Bypass loop !notATMCard, !isStolen, !accoundClosed, validPIN, declineFee

6 Same as path 1 up to the loop, Loop once !notATMCard, !isStolen, !accountClosed, !validPIN && n==1, validPIN && n==2, !declineFee

7 Same as path 1 up to the loop, Loop max times !notATMCard, !isStolen, !accountClosed, !validPIN && n==1, !validPIN && n==2, !validPIN && n==3, !validPIN

39


3. Identify Special CasesSequence diagrams do not include:• Exception handling paths

Covered by Controlled Exception Test

• Bindings to polymorphic servers• The number of paths grows exponentially when a

polymorphic server is several subclasses deep

40


• Reduce the risks for polymorphic servers!• Identify circumstances that must not occur and involve

polymorphic objects without direct interface• Identify constraints that must always be observed and

involve polymorphic objects without direct interface

• Develop test cases for the “must” and “must not” scenarios

41


4. Identify inputs and states• To identify test case values just look at the

path conditions for each path• For each path pick a set of input and state values that

satisfies the particular path conditions • In most cases there will exist many different sets of

values that will satisfy the path conditions!

• Finally determine the expected results for each test case

42


• Consequences Accurate sequence diagram required! Not everything is modeled

Additional tests required Round-trip path heuristic far more effective than

poking around Expansion of paths resulting from polymorphic

bindings not addressed.

43

Overview



44

Mode Machine Test

• Intent Develop a test suite for subsystem scope behavior

by analyzing component behavior• Modal subsystem

Dominant control object Implements a state pattern Behavior is sequentially constrained

• The same sequence of external events does not always produce the same response

> Example: vending machines

45

Mode Machine Test

• A correctly functioning subsystem must Accept legal events Reject events that are illegal Produce a correct resultant state Produce a correct action for each test event

• Five kinds of control faults Missing transition Incorrect transition Invalid resultant state Production of a corrupt state Sneak Path

46

Mode Machine Test

• Creating a Test Model Identify the dominant control object (DCO) and

develop its state model Identify the external events and actions Map the external events onto the DCO Develop a modal test suite for this state model

47

Mode Machine Test

• N+ Test Suite Conformance tests

• Regular tests

Guard expansion• Boolean outcomes

Iteration expansion• Each of the items gets selected once

Implicit transition expansion• Events accepted in every state

Sneak path analysis• Illegal sequences

48

Mode Machine Test

• Entry Criteria Each component has met the exit criteria for the

applicable test pattern Each cluster has minimal operability requirements

• Exit Criteria There is a test for every root-to-leaf path in the

expanded transition tree and each sneak path pair

Documents

Software Testing Chapter 12: Subsystems Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen