Integration Testing

Testing of Large Systems 1

Integration Testing


[Abrain and Moore, 2004]

Testing levels– Unit testing

• Verifies isolated functioning of seperately testable softwarepieces

– Integration testing• Verifying the interaction between software components• Architecture-driven

– System testing• Concerned with the behavior of the full system• Functional and ”non-functional”/quality properties of the

system


War Story I

In a project I was developing a component forcreating UML sequence diagrams– The component worked perfectly well (presentation,

editing, undo/redo, load/save, …) when tested byitself

When combined and tested with a class diagramcomponent it failed– The class diagram component might delete/change

packages related to sequence diagram• So if a class was created, sequence diagrams containing

objects instantiated from that class created, and the classdeleted this would lead to a failure


War Story II

I was writing a component that should interfacewith a serial port temperature sensing device(TH-03)– It would read a temperature based on a protocol for

TH-03, do some processing, and eventually give thattemperature away

Worked well… When connected to actual TH-03 device, nothing

happened– Needed to set special serial port options to power

device through serial cable…


Integration defects

Binder list typicalinterface defects…– “2/3 of defects are in

integration”

War stories arecovered by it


Integration testing

Integration testing deals with finding defects in the wayindividual parts work together

We use composition at almost any level in softwareengineering– May argue (as does Binder [2000]) that integration testing occurs

at all levels.


Why integration testing?

Why test component interactions???

If we provide– ‘sufficient’ unit testing

and– ‘sufficient’ system testing

is it not OK then???


Why integration testing?

Binder relates an argument (on axiomatizing test case adequacy):

Antidecomposition axiom:– There exists a program P and component Q [of P] such that T is

adequate for P, T’ is the set of vectors of values that variables canassume on entrance to Q for some t of T, and T’ is not adequate for Q

– System scope coverage does not necessarily achieve componentcoverage

– Why? Examples?

Anticomposition axiom:– There exist programs P and Q, and test set T, such that T is adequate

for P, and the set of vectors of values that variables can assume onentrance to Q for inputs in T is adequate for Q, but T is not adequate forP;Q

– Adequate testing at component level is not equivalent to adequatetesting of a system of components

– Why? Examples?


Premise for integration testing

At any given level: System consisting of components

Testing interactions between componentsrequire that these are stable– Stable = sufficient dependability to allow integration

testing– Threshold reflects practical considerations

In case of unstable component – then what?


Component stability/volatility

An important aspect is of course to preserve ourinvestment in testing

Spending staff hours to test a class that is laterfundamentally changed is a waste of time– In opposition to our wish for early integration testing– Postpone integration of volatile components!


Integration and architecture

Integration testing is highly coupled toarchitecture

Definition by [Bass et al., 2003]

The software architecture of a computing systemis the structures of the system, which comprise

software elements, the externally visibleproperties of those elements, and the

relationships among them.


Integration and architecture

Thus the architecture defines the testing in termsof what are

– the elements• (stable units that have already been unit tested adequately)

– and the relations• interactions that contain defects we wish to reveal

(Binder focuses on the module view of asoftware architecture)


Dependencies

Components may interact in a large number ofways using either explicit or implicit relations– composition and aggregation– delegation to objects / API calls– inheritance– global variables– instance variable access– objects as message parameters– RMI / socket communication


Dependencies

Many are however implicit dependencies thatare much harder to spot– database access / persistent store access– initialization sequences– timing constraints– file formatting / coding of byte streams

Binder states:– Integration testing does not focus on implicit

dependencies– ?


Dependency analysis

(Explicit) dependencies often dictate thesequence of testing– I.e. in which order are component interactions tested?

Thus a very helpful tool is to do a dependencyanalysis– Similar to what build systems do to determine

compilation order– Hard for implicit dependencies


Coupling

A measure of the strength of dependencies between twosubsystems

Which is preferable from a testing perspective?

Cf. testing tactics

high coupling low coupling


Example from Binder


Example from Binder

Remove cycles, do atopological sort on thedirected, acyclical dependencygraph

Root level– unit not used by any other unit

in cluster under test– often there are several roots

Leaf level– units that do not use any other

units in cluster under test. Cycles

– either be tested as a “unit”– or stubs introduced to break

cycle

Here arrows are uses relations


Lessons learned

Key lessons from early 1970s is that incremental integration and testing is most

effective

Integrating all components at the end ofdevelopment is problematic:– Last-minute changes are necessary but no time for

adequate testing– Testing often not systematic


Lessons learned

Advantages in incremental testing

– Interfaces systematically exercised and shown stablebefore new unproven interfaces are exercised

– Observed failures more likely to come from mostrecently added components making debugging moreefficient.


Stubs

So, the doctrine says integration test often buthow do we do integration with components thatare not stable or not developed at all?

Stub: partial or surrogate implementation of acomponent


Reasons for stubs

Stubs are powerful tools for many reasons

Make testing of certain conditions possible– Hardware simulator that outputs data that are seldom occurring

in practice• Example: specific patterns of temperature from temperature sensor

– Replace random behavior with predictable behavior• Example: testing backgammon move validation (calling

getNextMove() or similar to get move to validate)

Make testing economic– Stub for a database that is costly to reset/set up


Reasons for stubs

Often stubs are relevant to decouple interactions– Stub for complex algorithm with ‘simple’ answer

• Ensures that the defect found does not lie in the algorithmimplementation

– Stub for a component in a cycle

There is no free lunch!– Stubs can be costly to produce– … and sometimes you chase defects in production

code – that are actually embedded in the stubs

Morale: Keeps stubs simple!


Integration Strategies

Binder lists nine integration strategies,documented in pattern form, for doing integrationtesting

We focus on the classic strategies:– Big Bang (actually an anti-pattern )– Bottom-up– Top-down


Big Bang

Intent– demonstrate stability by attempting to exercise an

entire system with a few test runs Context

– bring all components together all at once. Allinterfaces tested in one go.

– usually ends in ‘big-bang’ – system dies miserably… Entry criteria

– all components have passed unit testing Exit criteria

– Test suite passes


Big Bang

Consequences– Fails – then what? Failure diagnosis is very difficult.– Even if exit criteria met, many interface faults can still

hide.– On the plus side: if it works, no effort has been spent

on test drivers and writing stubs.

– may be the course of action for• small systems with adequate unit testing• existing system with only minor modifications• system made from certified high quality reusable

components


Bottom Up

Intent– Demonstrate system stability by adding components

to the SUT in uses-dependency order, starting withcomponent having the fewest dependencies

Context– stepwise verification of tightly coupled components


Bottom up

Strategy: 1. stage: leafs


Bottom up

2. stage..


Bottom up

Last stage..


Bottom up

Entry criteria– components pass unit tests

Exit criteria– interface for each subcomponent has been exercised

at least once– complete when all root-level components pass test

suites


Bottom up

Consequences– Actually most unit tests in OO are partly integration tests

• Bottom-up testing implicitly takes place in a bottom up developmentprocess

Disadvantages– Driver development cost significant– Fix in lower level component may require revisions and retest up

the chain– Interfaces only indirectly exercised – only the ones used by the

component under test– Upper levels testing may require stubs in the bottom to test

special conditions– High level testing very late in cycle, i.e., validation only possible

late


Bottom up

Advantages– Parallel implementation and testing possible– Little need for stub writing


Tools: JUnit

Stubs?

Drivers?


Tools: Mock Objects

http://www.mockobjects.com/ /http://www.jmock.org/– Library for testing Java code using mock objects

Mock objects– Given an interface create an advanced stub at

runtime using reflection– May define expected values on mock objects using

constraints

Example...


Tools: Using jMock


Top Down

Intent– Demonstrate stability by

adding components to theSUT in control hierarchyorder, beginning with thetop-level control objects

Strategy– 1 stage:

• test control objects


Top Down

2. stage


Top Down

Final Stage


Top Down

Entry– each component to be integrated passes unit test

Exit– interface of each component has been exercised at

least once– complete when leaf-level components passes system

scope test suite


Top Down

Disadvantages– Large number of stubs necessary which is costly– Stubs are brittle– Fix in lower level component may require revisions up

the chain– Difficult to get lower level components sufficiently

exercised– Lower level components are exercised late

Advantages– Low driver development costs– early demonstration of user-related behavior


Variations

There are many variations of these strategies– Sandwich testing

• Moving from both top and bottom

Bottom line– there is no substitute from being clever and utilize a

combination of techniques that is most cost-efficientfor the project at hand.


High-frequency integration

Binder also describes high-frequency integrationthat is a strategy whose characteristics lies inthe timing, not the ordering, of testing– It is an intrinsic part of the process pattern daily build– At an extreme: continuous integration of eXtreme

Programming fame

Intent– Integrate new code with stabilized baseline frequently

to prevent integration bugs from going undiscoveredfor a long time; and to prevent divergence from thebaseline



Context– A stable baseline must be present; increments are the

focus of high-frequency integration– Increments size must match integration frequency

• daily builds + integrations -> increments must be deliverablein a day

– Test code developed in parallel with code– Testing must be automated…– Software Configuration Management must be in place



Procedure:

First– Revise code + test code on private branch– Desk check code and test– When all component testing passes, check-in to the integration

branch Second

– Integration tester builds system of increments– Testing using

• smoke tests and as much additional as time permits

Any increments that break HFI are correctedimmediately



Disadvantages– Automated tests must be in place– High commitment to maintaining code as well as tests– Be aware of adequacy criteria – the suite that found

the old bugs may not find the new ones Advantages

– Focus on maintaining tests is an effective bugprevention strategy

– Defects found early; debugging easier– Morale high as system works early and keeps doing it


Integration test in the TMM (1/2)

Basic integration test appears in level 2 (as does allbasic test techniques/methods).

Planning of the test process (incl. integration test) is alsoinitiated at level 2.

At level 3, integration test (and all other test activities) isintegrated in the sw life cycle.– In modern iterative/agile development methods, this happens

more or less automatically At level 4, the integration test process gets measured. At level 5 the integration test process is continuously

optimized.


Integration test in the TMM (2/2)

At level 2,At level 2, Managers

– Should ensure that multilevel testing is part of policies,planned for and catered for via project schedules andresources

Developers– Should help PM’s to make multilevel testing work, help

develop use cases/acceptance criteria, implement tests atall levels

Users/clients– Should help provide acceptance criteria, provide ample

and due feedback, participate in tests (accept/beta)


Summary

Testing levels– Unit testing

• Verifies isolated functioning of seperately testable softwarepieces

– Integration testing• Verifying the interaction between software components• Architecture-driven

– System testing• Concerned with the behavior of the full system• Functional and ”non-functional”/quality properties of the

system

Documents

Integration Testing